2021-03-24T20:35:41.365 INFO:root:teuthology version: 1.1.0-6b3150e9 2021-03-24T20:35:41.367 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi/5994655 --owner scheduled_sage@teuthology --description upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} --verbose --name sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi -- /home/teuthworker/archive/sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi/5994655/orig.config.yaml 2021-03-24T20:35:41.405 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-03-24T20:35:41.460 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi/5994655 branch: wip-sage-testing-2021-03-24-1139 description: upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary} email: null first_in_suite: false job_id: '5994655' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi nuke-on-error: true overrides: admin_socket: branch: wip-sage-testing-2021-03-24-1139 ceph: conf: client: debug rgw: 20 rgw crypt require ssl: false rgw crypt s3 kms backend: testing rgw crypt s3 kms encryption keys: testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= rgw curl low speed time: 300 rgw data log num shards: 4 rgw md log max shards: 4 rgw sync log trim interval: 0 rgw sync obj etag verify: true setgroup: ceph setuser: ceph mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: debug ms: 1 debug osd: 20 log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(PG_AVAILABILITY\) log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) sha1: 9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2 wait-for-scrub: false ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: branch: octopus sha1: 9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2 rgw: compression type: random frontend: beast rgw-multisite: realm: is default: true name: test-realm zonegroups: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zonegroup zones: - endpoints: - c1.client.0 is_default: true is_master: true name: test-zone1 - endpoints: - c2.client.0 is_default: true name: test-zone2 - endpoints: - c2.client.1 is_pubsub: true name: test-zone3 rgw-multisite-tests: args: - tests.py - tests_ps.py branch: octopus repo: https://github.com/ceph/ceph.git workunit: branch: wip-sage-testing-2021-03-24-1139 sha1: 9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2 owner: scheduled_sage@teuthology priority: 75 repo: git://git.ceph.com/ceph-ci.git roles: - - c1.mon.a - c1.mgr.x - c1.osd.0 - c1.osd.1 - c1.osd.2 - c1.client.0 - - c2.mon.a - c2.mgr.x - c2.osd.0 - c2.osd.1 - c2.osd.2 - c2.client.0 - c2.client.1 sha1: 9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2 sleep_before_teardown: 0 suite: upgrade:octopus-x suite_branch: wip-sage-testing-2021-03-24-1139 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2 targets: smithi052.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdmeEuI/ElAp8wa3lVeLfXYahy/pX0c/EdGRCcDqpaRhyuHrjHU0PEe8ckqbf14SelozcIfJk0CStTIKUBckWCPM5LZwtXt7Vh7NQCk6AHxWQq1egfGetVeWbzuPWS+q+doQ+4EyNnnZyWjnIXPtJc4R4tr/mCnKYWRQqax2ypu7oGzF/XCnsQeiRDdY1LWNxobRIpSIEGCJfD4NgJ1MSC4TrgjAw95hrAZgI480tzXReYyKL+WsdNrH/M4fVdlNXXNn5InUbzLyLYKaUfOkbTzr0Jp6SyzBKkBzcCg7kivZwOCv/HNZuHjh3ToUWZValSR3cjBMiSadmLcUBWcQNN smithi063.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJpxVWP+MN/3YlQybedqnQLiHHtDwzkWm82LhqQYZMkhGsE3UDKFv4wmJiTlzH9DcdOW55ds5k0h8KWkxBwtGUf2Rpz3o1MYiLkHk7T/pSG1CC02Iwy7Mt8Je0N3HebLVwXGmBUDu+VIuo8CZNk3dW2KYmWISpqWkywv3/IsA4aMBtRnpYGISapE0Y67MEsM7thcPSAtroKsTa4rZnYbhySBlufPLGKlHn10I53dR5brooEVN4sfHXKATbKSUmG7DxsJOWhvu3zzR+3RSZ/aS8r40WQE5wdKjMt6n6MASnDB2nX/ztpeZK1EebhPzMoK6EhSudrfaq/cOGBB2y7bTL tasks: - install: null - ceph: cluster: c1 - ceph: cluster: c2 - parallel: - upgrade-task - rgw: c1.client.0: port: 8000 c2.client.0: port: 8000 c2.client.1: port: 8001 - rgw-multisite: null - rgw-multisite-tests: config: reconfigure_delay: 60 teuthology_branch: master teuthology_sha1: 6b3150e9e0aa7ca432e26f31d87920ebd77f3708 timestamp: 2021-03-24_19:49:41 tube: smithi upgrade-task: - install.upgrade: c2.client.0: null c2.client.1: null c2.mgr.x: null c2.mon.a: null c2.osd.0: null c2.osd.1: null c2.osd.2: null user: sage verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.9855 2021-03-24T20:35:41.460 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2/qa; will attempt to use it 2021-03-24T20:35:41.461 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2/qa/tasks 2021-03-24T20:35:41.462 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-03-24T20:35:41.464 INFO:teuthology.task.internal:Checking packages... 2021-03-24T20:35:41.464 INFO:teuthology.task.internal:Checking packages skipped, missing os_type 'None' or ceph hash '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2' 2021-03-24T20:35:41.464 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-03-24T20:35:41.473 INFO:teuthology.task.internal:no buildpackages task found 2021-03-24T20:35:41.473 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-03-24T20:35:41.561 INFO:teuthology.task.internal:Saving configuration 2021-03-24T20:35:41.575 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-03-24T20:35:41.608 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-24T20:35:41.626 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-24 20:30:17.140919', 'description': '/home/teuthworker/archive/sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi/5994655', 'locked_by': 'scheduled_sage@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIGtxQx7GAIb3QNvEqLXT4ZrIspkWqv+u8bDdZNGckbKY', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi052.front.sepia.ceph.com'} 2021-03-24T20:35:41.644 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-24 20:30:17.140899', 'description': '/home/teuthworker/archive/sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi/5994655', 'locked_by': 'scheduled_sage@teuthology', 'up': True, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAICWoWN802xxxEwNPn3JzxodmbAfw4JAcD+SW5E0vyjMU', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi063.front.sepia.ceph.com'} 2021-03-24T20:35:41.645 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-03-24T20:35:41.657 INFO:teuthology.task.internal:roles: ubuntu@smithi052.front.sepia.ceph.com - ['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0'] 2021-03-24T20:35:41.658 INFO:teuthology.task.internal:roles: ubuntu@smithi063.front.sepia.ceph.com - ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1'] 2021-03-24T20:35:41.658 INFO:teuthology.run_tasks:Running task console_log... 2021-03-24T20:35:41.772 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb71ecbc9d8>, signals=[15]) 2021-03-24T20:35:41.774 INFO:teuthology.run_tasks:Running task internal.connect... 2021-03-24T20:35:41.852 INFO:teuthology.task.internal:Opening connections... 2021-03-24T20:35:41.853 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi052.front.sepia.ceph.com 2021-03-24T20:35:41.854 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T20:35:41.927 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi063.front.sepia.ceph.com 2021-03-24T20:35:41.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T20:35:42.000 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-03-24T20:35:42.011 DEBUG:teuthology.orchestra.run.smithi052:> uname -m 2021-03-24T20:35:42.092 INFO:teuthology.orchestra.run.smithi052.stdout:x86_64 2021-03-24T20:35:42.093 DEBUG:teuthology.orchestra.run.smithi052:> cat /etc/os-release 2021-03-24T20:35:42.140 INFO:teuthology.orchestra.run.smithi052.stdout:NAME="Ubuntu" 2021-03-24T20:35:42.141 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-03-24T20:35:42.141 INFO:teuthology.orchestra.run.smithi052.stdout:ID=ubuntu 2021-03-24T20:35:42.141 INFO:teuthology.orchestra.run.smithi052.stdout:ID_LIKE=debian 2021-03-24T20:35:42.141 INFO:teuthology.orchestra.run.smithi052.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-03-24T20:35:42.141 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_ID="18.04" 2021-03-24T20:35:42.142 INFO:teuthology.orchestra.run.smithi052.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-24T20:35:42.142 INFO:teuthology.orchestra.run.smithi052.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-24T20:35:42.142 INFO:teuthology.orchestra.run.smithi052.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-24T20:35:42.142 INFO:teuthology.orchestra.run.smithi052.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-24T20:35:42.143 INFO:teuthology.orchestra.run.smithi052.stdout:VERSION_CODENAME=bionic 2021-03-24T20:35:42.143 INFO:teuthology.orchestra.run.smithi052.stdout:UBUNTU_CODENAME=bionic 2021-03-24T20:35:42.144 INFO:teuthology.lock.ops:Updating smithi052.front.sepia.ceph.com on lock server 2021-03-24T20:35:42.160 DEBUG:teuthology.orchestra.run.smithi063:> uname -m 2021-03-24T20:35:42.191 INFO:teuthology.orchestra.run.smithi063.stdout:x86_64 2021-03-24T20:35:42.192 DEBUG:teuthology.orchestra.run.smithi063:> cat /etc/os-release 2021-03-24T20:35:42.238 INFO:teuthology.orchestra.run.smithi063.stdout:NAME="Ubuntu" 2021-03-24T20:35:42.239 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION="18.04.5 LTS (Bionic Beaver)" 2021-03-24T20:35:42.239 INFO:teuthology.orchestra.run.smithi063.stdout:ID=ubuntu 2021-03-24T20:35:42.239 INFO:teuthology.orchestra.run.smithi063.stdout:ID_LIKE=debian 2021-03-24T20:35:42.239 INFO:teuthology.orchestra.run.smithi063.stdout:PRETTY_NAME="Ubuntu 18.04.5 LTS" 2021-03-24T20:35:42.239 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_ID="18.04" 2021-03-24T20:35:42.240 INFO:teuthology.orchestra.run.smithi063.stdout:HOME_URL="https://www.ubuntu.com/" 2021-03-24T20:35:42.240 INFO:teuthology.orchestra.run.smithi063.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2021-03-24T20:35:42.240 INFO:teuthology.orchestra.run.smithi063.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2021-03-24T20:35:42.240 INFO:teuthology.orchestra.run.smithi063.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2021-03-24T20:35:42.241 INFO:teuthology.orchestra.run.smithi063.stdout:VERSION_CODENAME=bionic 2021-03-24T20:35:42.241 INFO:teuthology.orchestra.run.smithi063.stdout:UBUNTU_CODENAME=bionic 2021-03-24T20:35:42.241 INFO:teuthology.lock.ops:Updating smithi063.front.sepia.ceph.com on lock server 2021-03-24T20:35:42.256 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-03-24T20:35:42.270 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-03-24T20:35:42.300 INFO:teuthology.task.internal:Checking for old test directory... 2021-03-24T20:35:42.301 DEBUG:teuthology.orchestra.run.smithi052:> test '!' -e /home/ubuntu/cephtest 2021-03-24T20:35:42.304 DEBUG:teuthology.orchestra.run.smithi063:> test '!' -e /home/ubuntu/cephtest 2021-03-24T20:35:42.309 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-03-24T20:35:42.344 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-03-24T20:35:42.344 DEBUG:teuthology.orchestra.run.smithi052:> test -z $(ls -A /var/lib/ceph) 2021-03-24T20:35:42.352 DEBUG:teuthology.orchestra.run.smithi063:> test -z $(ls -A /var/lib/ceph) 2021-03-24T20:35:42.362 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-03-24T20:35:42.467 INFO:teuthology.run_tasks:Running task kernel... 2021-03-24T20:35:42.486 DEBUG:teuthology.task.kernel:normalized config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c1.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c1.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c1.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mgr.x': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.0': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.1': {'kdb': True, 'sha1': 'distro'}, 'c2.osd.2': {'kdb': True, 'sha1': 'distro'}, 'c2.client.0': {'kdb': True, 'sha1': 'distro'}, 'c2.client.1': {'kdb': True, 'sha1': 'distro'}} 2021-03-24T20:35:42.486 INFO:teuthology.task.kernel:config {'c1.mon.a': {'kdb': True, 'sha1': 'distro'}, 'c2.mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-03-24T20:35:42.487 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2021-03-24T20:35:42.493 INFO:teuthology.orchestra.run.smithi052.stdout:4.15.0-136-generic 2021-03-24T20:35:42.494 INFO:teuthology.task.kernel:Running kernel on smithi052: 4.15.0-136-generic 2021-03-24T20:35:42.494 DEBUG:teuthology.orchestra.run.smithi052:> sudo apt-get clean 2021-03-24T20:35:42.611 DEBUG:teuthology.orchestra.run.smithi052:> sudo apt-get update 2021-03-24T20:35:42.889 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-24T20:35:42.890 INFO:teuthology.orchestra.run.smithi052.stdout:Get:2 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-03-24T20:35:42.952 INFO:teuthology.orchestra.run.smithi052.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-03-24T20:35:43.244 INFO:teuthology.orchestra.run.smithi052.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-03-24T20:35:43.339 INFO:teuthology.orchestra.run.smithi052.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [946 kB] 2021-03-24T20:35:43.474 INFO:teuthology.orchestra.run.smithi052.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,248 kB] 2021-03-24T20:35:43.736 INFO:teuthology.orchestra.run.smithi052.stdout:Get:7 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,627 kB] 2021-03-24T20:35:43.847 INFO:teuthology.orchestra.run.smithi052.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,967 kB] 2021-03-24T20:35:43.885 INFO:teuthology.orchestra.run.smithi052.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [309 kB] 2021-03-24T20:35:43.888 INFO:teuthology.orchestra.run.smithi052.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [261 kB] 2021-03-24T20:35:43.899 INFO:teuthology.orchestra.run.smithi052.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [17.7 kB] 2021-03-24T20:35:43.900 INFO:teuthology.orchestra.run.smithi052.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [34.1 kB] 2021-03-24T20:35:43.902 INFO:teuthology.orchestra.run.smithi052.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,116 kB] 2021-03-24T20:35:43.965 INFO:teuthology.orchestra.run.smithi052.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [400 kB] 2021-03-24T20:35:43.976 INFO:teuthology.orchestra.run.smithi052.stdout:Get:15 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [979 kB] 2021-03-24T20:35:44.007 INFO:teuthology.orchestra.run.smithi052.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [282 kB] 2021-03-24T20:35:44.015 INFO:teuthology.orchestra.run.smithi052.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [24.3 kB] 2021-03-24T20:35:44.016 INFO:teuthology.orchestra.run.smithi052.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [37.7 kB] 2021-03-24T20:35:44.017 INFO:teuthology.orchestra.run.smithi052.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,562 kB] 2021-03-24T20:35:44.037 INFO:teuthology.orchestra.run.smithi052.stdout:Get:20 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [250 kB] 2021-03-24T20:35:44.043 INFO:teuthology.orchestra.run.smithi052.stdout:Get:21 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,480 B] 2021-03-24T20:35:44.043 INFO:teuthology.orchestra.run.smithi052.stdout:Get:22 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [19.1 kB] 2021-03-24T20:35:44.044 INFO:teuthology.orchestra.run.smithi052.stdout:Get:23 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [4,412 B] 2021-03-24T20:35:44.055 INFO:teuthology.orchestra.run.smithi052.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,723 kB] 2021-03-24T20:35:44.130 INFO:teuthology.orchestra.run.smithi052.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [365 kB] 2021-03-24T20:35:44.135 INFO:teuthology.orchestra.run.smithi052.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [11.6 kB] 2021-03-24T20:35:44.135 INFO:teuthology.orchestra.run.smithi052.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [24.9 kB] 2021-03-24T20:35:44.136 INFO:teuthology.orchestra.run.smithi052.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,464 B] 2021-03-24T20:35:45.763 INFO:teuthology.orchestra.run.smithi052.stdout:Fetched 13.5 MB in 3s (4,613 kB/s) 2021-03-24T20:35:47.306 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2021-03-24T20:35:47.354 DEBUG:teuthology.orchestra.run.smithi052:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-24T20:35:47.434 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2021-03-24T20:35:47.677 INFO:teuthology.orchestra.run.smithi052.stdout:Building dependency tree... 2021-03-24T20:35:47.678 INFO:teuthology.orchestra.run.smithi052.stdout:Reading state information... 2021-03-24T20:35:47.764 INFO:teuthology.orchestra.run.smithi052.stderr:E: Unable to locate package linux-image-current-generic 2021-03-24T20:35:47.765 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-24T20:35:47.766 DEBUG:teuthology.orchestra.run.smithi052:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-24T20:35:47.846 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2021-03-24T20:35:48.093 INFO:teuthology.orchestra.run.smithi052.stdout:Building dependency tree... 2021-03-24T20:35:48.094 INFO:teuthology.orchestra.run.smithi052.stdout:Reading state information... 2021-03-24T20:35:48.283 INFO:teuthology.orchestra.run.smithi052.stdout:The following additional packages will be installed: 2021-03-24T20:35:48.283 INFO:teuthology.orchestra.run.smithi052.stdout: linux-generic linux-headers-4.15.0-139 linux-headers-4.15.0-139-generic 2021-03-24T20:35:48.283 INFO:teuthology.orchestra.run.smithi052.stdout: linux-headers-generic linux-image-4.15.0-139-generic 2021-03-24T20:35:48.284 INFO:teuthology.orchestra.run.smithi052.stdout: linux-modules-4.15.0-139-generic linux-modules-extra-4.15.0-139-generic 2021-03-24T20:35:48.285 INFO:teuthology.orchestra.run.smithi052.stdout:Suggested packages: 2021-03-24T20:35:48.285 INFO:teuthology.orchestra.run.smithi052.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2021-03-24T20:35:48.285 INFO:teuthology.orchestra.run.smithi052.stdout:Recommended packages: 2021-03-24T20:35:48.286 INFO:teuthology.orchestra.run.smithi052.stdout: thermald 2021-03-24T20:35:48.316 INFO:teuthology.orchestra.run.smithi052.stdout:The following NEW packages will be installed: 2021-03-24T20:35:48.317 INFO:teuthology.orchestra.run.smithi052.stdout: linux-headers-4.15.0-139 linux-headers-4.15.0-139-generic 2021-03-24T20:35:48.317 INFO:teuthology.orchestra.run.smithi052.stdout: linux-image-4.15.0-139-generic linux-modules-4.15.0-139-generic 2021-03-24T20:35:48.318 INFO:teuthology.orchestra.run.smithi052.stdout: linux-modules-extra-4.15.0-139-generic 2021-03-24T20:35:48.318 INFO:teuthology.orchestra.run.smithi052.stdout:The following packages will be upgraded: 2021-03-24T20:35:48.319 INFO:teuthology.orchestra.run.smithi052.stdout: linux-generic linux-headers-generic linux-image-generic 2021-03-24T20:35:48.499 INFO:teuthology.orchestra.run.smithi052.stdout:3 upgraded, 5 newly installed, 0 to remove and 30 not upgraded. 2021-03-24T20:35:48.500 INFO:teuthology.orchestra.run.smithi052.stdout:Need to get 66.0 MB of archives. 2021-03-24T20:35:48.500 INFO:teuthology.orchestra.run.smithi052.stdout:After this operation, 334 MB of additional disk space will be used. 2021-03-24T20:35:48.500 INFO:teuthology.orchestra.run.smithi052.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-139-generic amd64 4.15.0-139.143 [13.1 MB] 2021-03-24T20:35:49.503 INFO:teuthology.orchestra.run.smithi052.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-139-generic amd64 4.15.0-139.143 [8,021 kB] 2021-03-24T20:35:49.746 INFO:teuthology.orchestra.run.smithi052.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-139-generic amd64 4.15.0-139.143 [32.8 MB] 2021-03-24T20:35:50.709 INFO:teuthology.orchestra.run.smithi052.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.139.126 [1,868 B] 2021-03-24T20:35:50.709 INFO:teuthology.orchestra.run.smithi052.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.139.126 [2,572 B] 2021-03-24T20:35:50.710 INFO:teuthology.orchestra.run.smithi052.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-139 all 4.15.0-139.143 [10.9 MB] 2021-03-24T20:35:51.022 INFO:teuthology.orchestra.run.smithi052.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-139-generic amd64 4.15.0-139.143 [1,101 kB] 2021-03-24T20:35:51.076 INFO:teuthology.orchestra.run.smithi052.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.139.126 [2,468 B] 2021-03-24T20:35:51.538 INFO:teuthology.orchestra.run.smithi052.stdout:Fetched 66.0 MB in 3s (24.1 MB/s) 2021-03-24T20:35:51.697 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package linux-modules-4.15.0-139-generic. 2021-03-24T20:35:51.734 INFO:teuthology.orchestra.run.smithi052.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102065 files and directories currently installed.) 2021-03-24T20:35:51.737 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../0-linux-modules-4.15.0-139-generic_4.15.0-139.143_amd64.deb ... 2021-03-24T20:35:51.786 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking linux-modules-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:35:53.520 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package linux-image-4.15.0-139-generic. 2021-03-24T20:35:53.531 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../1-linux-image-4.15.0-139-generic_4.15.0-139.143_amd64.deb ... 2021-03-24T20:35:53.626 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking linux-image-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:35:54.081 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-139-generic. 2021-03-24T20:35:54.093 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-139-generic_4.15.0-139.143_amd64.deb ... 2021-03-24T20:35:54.122 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking linux-modules-extra-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:35:57.787 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../3-linux-generic_4.15.0.139.126_amd64.deb ... 2021-03-24T20:35:57.908 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking linux-generic (4.15.0.139.126) over (4.15.0.136.123) ... 2021-03-24T20:35:58.224 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.139.126_amd64.deb ... 2021-03-24T20:35:58.345 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking linux-image-generic (4.15.0.139.126) over (4.15.0.136.123) ... 2021-03-24T20:35:58.665 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package linux-headers-4.15.0-139. 2021-03-24T20:35:58.678 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../5-linux-headers-4.15.0-139_4.15.0-139.143_all.deb ... 2021-03-24T20:35:58.715 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking linux-headers-4.15.0-139 (4.15.0-139.143) ... 2021-03-24T20:36:03.671 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package linux-headers-4.15.0-139-generic. 2021-03-24T20:36:03.692 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../6-linux-headers-4.15.0-139-generic_4.15.0-139.143_amd64.deb ... 2021-03-24T20:36:03.749 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking linux-headers-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:36:06.252 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.139.126_amd64.deb ... 2021-03-24T20:36:06.349 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking linux-headers-generic (4.15.0.139.126) over (4.15.0.136.123) ... 2021-03-24T20:36:06.706 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up linux-modules-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:36:07.254 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up linux-headers-4.15.0-139 (4.15.0-139.143) ... 2021-03-24T20:36:07.380 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up linux-headers-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:36:07.506 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up linux-image-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:36:07.975 INFO:teuthology.orchestra.run.smithi052.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-139-generic 2021-03-24T20:36:07.976 INFO:teuthology.orchestra.run.smithi052.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-139-generic 2021-03-24T20:36:08.109 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up linux-modules-extra-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:36:08.685 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up linux-headers-generic (4.15.0.139.126) ... 2021-03-24T20:36:08.811 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up linux-image-generic (4.15.0.139.126) ... 2021-03-24T20:36:08.937 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up linux-generic (4.15.0.139.126) ... 2021-03-24T20:36:09.063 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for linux-image-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:36:09.264 INFO:teuthology.orchestra.run.smithi052.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-03-24T20:36:09.264 INFO:teuthology.orchestra.run.smithi052.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-139-generic 2021-03-24T20:36:24.379 INFO:teuthology.orchestra.run.smithi052.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-03-24T20:36:24.379 INFO:teuthology.orchestra.run.smithi052.stdout:Sourcing file `/etc/default/grub' 2021-03-24T20:36:24.411 INFO:teuthology.orchestra.run.smithi052.stdout:Generating grub configuration file ... 2021-03-24T20:36:24.808 INFO:teuthology.orchestra.run.smithi052.stdout:Found linux image: /boot/vmlinuz-4.15.0-139-generic 2021-03-24T20:36:24.841 INFO:teuthology.orchestra.run.smithi052.stdout:Found initrd image: /boot/initrd.img-4.15.0-139-generic 2021-03-24T20:36:25.207 INFO:teuthology.orchestra.run.smithi052.stdout:Found linux image: /boot/vmlinuz-4.15.0-136-generic 2021-03-24T20:36:25.240 INFO:teuthology.orchestra.run.smithi052.stdout:Found initrd image: /boot/initrd.img-4.15.0-136-generic 2021-03-24T20:36:26.767 INFO:teuthology.orchestra.run.smithi052.stdout:done 2021-03-24T20:36:27.161 DEBUG:teuthology.orchestra.run.smithi052:> dpkg -s linux-image-generic 2021-03-24T20:36:27.230 INFO:teuthology.orchestra.run.smithi052.stdout:Package: linux-image-generic 2021-03-24T20:36:27.230 INFO:teuthology.orchestra.run.smithi052.stdout:Status: install ok installed 2021-03-24T20:36:27.231 INFO:teuthology.orchestra.run.smithi052.stdout:Priority: optional 2021-03-24T20:36:27.231 INFO:teuthology.orchestra.run.smithi052.stdout:Section: kernel 2021-03-24T20:36:27.231 INFO:teuthology.orchestra.run.smithi052.stdout:Installed-Size: 17 2021-03-24T20:36:27.232 INFO:teuthology.orchestra.run.smithi052.stdout:Maintainer: Ubuntu Kernel Team 2021-03-24T20:36:27.232 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: amd64 2021-03-24T20:36:27.232 INFO:teuthology.orchestra.run.smithi052.stdout:Source: linux-meta 2021-03-24T20:36:27.232 INFO:teuthology.orchestra.run.smithi052.stdout:Version: 4.15.0.139.126 2021-03-24T20:36:27.232 INFO:teuthology.orchestra.run.smithi052.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-03-24T20:36:27.233 INFO:teuthology.orchestra.run.smithi052.stdout:Depends: linux-image-4.15.0-139-generic, linux-modules-extra-4.15.0-139-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-24T20:36:27.233 INFO:teuthology.orchestra.run.smithi052.stdout:Recommends: thermald 2021-03-24T20:36:27.233 INFO:teuthology.orchestra.run.smithi052.stdout:Description: Generic Linux kernel image 2021-03-24T20:36:27.233 INFO:teuthology.orchestra.run.smithi052.stdout: This package will always depend on the latest generic kernel image 2021-03-24T20:36:27.234 INFO:teuthology.orchestra.run.smithi052.stdout: available. 2021-03-24T20:36:27.234 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-136-generic Expected: 4.15.0-139-generic 2021-03-24T20:36:27.234 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2021-03-24T20:36:27.241 INFO:teuthology.orchestra.run.smithi063.stdout:4.15.0-136-generic 2021-03-24T20:36:27.242 INFO:teuthology.task.kernel:Running kernel on smithi063: 4.15.0-136-generic 2021-03-24T20:36:27.242 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-get clean 2021-03-24T20:36:27.353 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-get update 2021-03-24T20:36:27.628 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:1 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-24T20:36:27.690 INFO:teuthology.orchestra.run.smithi063.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB] 2021-03-24T20:36:27.715 INFO:teuthology.orchestra.run.smithi063.stdout:Get:3 http://security.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB] 2021-03-24T20:36:28.060 INFO:teuthology.orchestra.run.smithi063.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-03-24T20:36:28.144 INFO:teuthology.orchestra.run.smithi063.stdout:Get:5 http://security.ubuntu.com/ubuntu bionic-security/main amd64 Packages [1,627 kB] 2021-03-24T20:36:28.269 INFO:teuthology.orchestra.run.smithi063.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main i386 Packages [1,248 kB] 2021-03-24T20:36:28.684 INFO:teuthology.orchestra.run.smithi063.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 Packages [1,967 kB] 2021-03-24T20:36:28.714 INFO:teuthology.orchestra.run.smithi063.stdout:Get:8 http://security.ubuntu.com/ubuntu bionic-security/main i386 Packages [946 kB] 2021-03-24T20:36:28.766 INFO:teuthology.orchestra.run.smithi063.stdout:Get:9 http://security.ubuntu.com/ubuntu bionic-security/main Translation-en [309 kB] 2021-03-24T20:36:28.799 INFO:teuthology.orchestra.run.smithi063.stdout:Get:10 http://security.ubuntu.com/ubuntu bionic-security/restricted amd64 Packages [261 kB] 2021-03-24T20:36:28.815 INFO:teuthology.orchestra.run.smithi063.stdout:Get:11 http://security.ubuntu.com/ubuntu bionic-security/restricted i386 Packages [17.7 kB] 2021-03-24T20:36:28.816 INFO:teuthology.orchestra.run.smithi063.stdout:Get:12 http://security.ubuntu.com/ubuntu bionic-security/restricted Translation-en [34.1 kB] 2021-03-24T20:36:28.816 INFO:teuthology.orchestra.run.smithi063.stdout:Get:13 http://security.ubuntu.com/ubuntu bionic-security/universe amd64 Packages [1,116 kB] 2021-03-24T20:36:28.820 INFO:teuthology.orchestra.run.smithi063.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic-updates/main Translation-en [400 kB] 2021-03-24T20:36:28.832 INFO:teuthology.orchestra.run.smithi063.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic-updates/restricted amd64 Packages [282 kB] 2021-03-24T20:36:28.867 INFO:teuthology.orchestra.run.smithi063.stdout:Get:16 http://security.ubuntu.com/ubuntu bionic-security/universe i386 Packages [979 kB] 2021-03-24T20:36:28.894 INFO:teuthology.orchestra.run.smithi063.stdout:Get:17 http://security.ubuntu.com/ubuntu bionic-security/universe Translation-en [250 kB] 2021-03-24T20:36:28.900 INFO:teuthology.orchestra.run.smithi063.stdout:Get:18 http://security.ubuntu.com/ubuntu bionic-security/multiverse amd64 Packages [19.1 kB] 2021-03-24T20:36:28.900 INFO:teuthology.orchestra.run.smithi063.stdout:Get:19 http://security.ubuntu.com/ubuntu bionic-security/multiverse i386 Packages [6,480 B] 2021-03-24T20:36:28.901 INFO:teuthology.orchestra.run.smithi063.stdout:Get:20 http://security.ubuntu.com/ubuntu bionic-security/multiverse Translation-en [4,412 B] 2021-03-24T20:36:28.919 INFO:teuthology.orchestra.run.smithi063.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic-updates/restricted i386 Packages [24.3 kB] 2021-03-24T20:36:28.920 INFO:teuthology.orchestra.run.smithi063.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic-updates/restricted Translation-en [37.7 kB] 2021-03-24T20:36:28.921 INFO:teuthology.orchestra.run.smithi063.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic-updates/universe i386 Packages [1,562 kB] 2021-03-24T20:36:28.938 INFO:teuthology.orchestra.run.smithi063.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 Packages [1,723 kB] 2021-03-24T20:36:29.003 INFO:teuthology.orchestra.run.smithi063.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic-updates/universe Translation-en [365 kB] 2021-03-24T20:36:29.010 INFO:teuthology.orchestra.run.smithi063.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse i386 Packages [11.6 kB] 2021-03-24T20:36:29.010 INFO:teuthology.orchestra.run.smithi063.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse amd64 Packages [24.9 kB] 2021-03-24T20:36:29.010 INFO:teuthology.orchestra.run.smithi063.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic-updates/multiverse Translation-en [6,464 B] 2021-03-24T20:36:30.510 INFO:teuthology.orchestra.run.smithi063.stdout:Fetched 13.5 MB in 3s (4,588 kB/s) 2021-03-24T20:36:31.950 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2021-03-24T20:36:31.998 DEBUG:teuthology.orchestra.run.smithi063:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-24T20:36:32.078 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2021-03-24T20:36:32.320 INFO:teuthology.orchestra.run.smithi063.stdout:Building dependency tree... 2021-03-24T20:36:32.320 INFO:teuthology.orchestra.run.smithi063.stdout:Reading state information... 2021-03-24T20:36:32.406 INFO:teuthology.orchestra.run.smithi063.stderr:E: Unable to locate package linux-image-current-generic 2021-03-24T20:36:32.408 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-24T20:36:32.409 DEBUG:teuthology.orchestra.run.smithi063:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-24T20:36:32.491 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2021-03-24T20:36:32.749 INFO:teuthology.orchestra.run.smithi063.stdout:Building dependency tree... 2021-03-24T20:36:32.750 INFO:teuthology.orchestra.run.smithi063.stdout:Reading state information... 2021-03-24T20:36:32.942 INFO:teuthology.orchestra.run.smithi063.stdout:The following additional packages will be installed: 2021-03-24T20:36:32.942 INFO:teuthology.orchestra.run.smithi063.stdout: linux-generic linux-headers-4.15.0-139 linux-headers-4.15.0-139-generic 2021-03-24T20:36:32.942 INFO:teuthology.orchestra.run.smithi063.stdout: linux-headers-generic linux-image-4.15.0-139-generic 2021-03-24T20:36:32.943 INFO:teuthology.orchestra.run.smithi063.stdout: linux-modules-4.15.0-139-generic linux-modules-extra-4.15.0-139-generic 2021-03-24T20:36:32.943 INFO:teuthology.orchestra.run.smithi063.stdout:Suggested packages: 2021-03-24T20:36:32.943 INFO:teuthology.orchestra.run.smithi063.stdout: fdutils linux-doc-4.15.0 | linux-source-4.15.0 linux-tools 2021-03-24T20:36:32.944 INFO:teuthology.orchestra.run.smithi063.stdout:Recommended packages: 2021-03-24T20:36:32.944 INFO:teuthology.orchestra.run.smithi063.stdout: thermald 2021-03-24T20:36:32.977 INFO:teuthology.orchestra.run.smithi063.stdout:The following NEW packages will be installed: 2021-03-24T20:36:32.978 INFO:teuthology.orchestra.run.smithi063.stdout: linux-headers-4.15.0-139 linux-headers-4.15.0-139-generic 2021-03-24T20:36:32.978 INFO:teuthology.orchestra.run.smithi063.stdout: linux-image-4.15.0-139-generic linux-modules-4.15.0-139-generic 2021-03-24T20:36:32.979 INFO:teuthology.orchestra.run.smithi063.stdout: linux-modules-extra-4.15.0-139-generic 2021-03-24T20:36:32.979 INFO:teuthology.orchestra.run.smithi063.stdout:The following packages will be upgraded: 2021-03-24T20:36:32.980 INFO:teuthology.orchestra.run.smithi063.stdout: linux-generic linux-headers-generic linux-image-generic 2021-03-24T20:36:33.159 INFO:teuthology.orchestra.run.smithi063.stdout:3 upgraded, 5 newly installed, 0 to remove and 30 not upgraded. 2021-03-24T20:36:33.159 INFO:teuthology.orchestra.run.smithi063.stdout:Need to get 66.0 MB of archives. 2021-03-24T20:36:33.159 INFO:teuthology.orchestra.run.smithi063.stdout:After this operation, 334 MB of additional disk space will be used. 2021-03-24T20:36:33.160 INFO:teuthology.orchestra.run.smithi063.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-4.15.0-139-generic amd64 4.15.0-139.143 [13.1 MB] 2021-03-24T20:36:34.158 INFO:teuthology.orchestra.run.smithi063.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-4.15.0-139-generic amd64 4.15.0-139.143 [8,021 kB] 2021-03-24T20:36:34.358 INFO:teuthology.orchestra.run.smithi063.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-modules-extra-4.15.0-139-generic amd64 4.15.0-139.143 [32.8 MB] 2021-03-24T20:36:35.376 INFO:teuthology.orchestra.run.smithi063.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-generic amd64 4.15.0.139.126 [1,868 B] 2021-03-24T20:36:35.376 INFO:teuthology.orchestra.run.smithi063.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-image-generic amd64 4.15.0.139.126 [2,572 B] 2021-03-24T20:36:35.377 INFO:teuthology.orchestra.run.smithi063.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-139 all 4.15.0-139.143 [10.9 MB] 2021-03-24T20:36:35.676 INFO:teuthology.orchestra.run.smithi063.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-4.15.0-139-generic amd64 4.15.0-139.143 [1,101 kB] 2021-03-24T20:36:35.720 INFO:teuthology.orchestra.run.smithi063.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 linux-headers-generic amd64 4.15.0.139.126 [2,468 B] 2021-03-24T20:36:36.178 INFO:teuthology.orchestra.run.smithi063.stdout:Fetched 66.0 MB in 3s (24.2 MB/s) 2021-03-24T20:36:36.342 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package linux-modules-4.15.0-139-generic. 2021-03-24T20:36:36.381 INFO:teuthology.orchestra.run.smithi063.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102065 files and directories currently installed.) 2021-03-24T20:36:36.383 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../0-linux-modules-4.15.0-139-generic_4.15.0-139.143_amd64.deb ... 2021-03-24T20:36:36.422 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-modules-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:36:38.181 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package linux-image-4.15.0-139-generic. 2021-03-24T20:36:38.194 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../1-linux-image-4.15.0-139-generic_4.15.0-139.143_amd64.deb ... 2021-03-24T20:36:38.285 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-image-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:36:38.776 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package linux-modules-extra-4.15.0-139-generic. 2021-03-24T20:36:38.787 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../2-linux-modules-extra-4.15.0-139-generic_4.15.0-139.143_amd64.deb ... 2021-03-24T20:36:38.826 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-modules-extra-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:36:42.508 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../3-linux-generic_4.15.0.139.126_amd64.deb ... 2021-03-24T20:36:42.612 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-generic (4.15.0.139.126) over (4.15.0.136.123) ... 2021-03-24T20:36:42.928 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../4-linux-image-generic_4.15.0.139.126_amd64.deb ... 2021-03-24T20:36:43.049 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-image-generic (4.15.0.139.126) over (4.15.0.136.123) ... 2021-03-24T20:36:43.369 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package linux-headers-4.15.0-139. 2021-03-24T20:36:43.382 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../5-linux-headers-4.15.0-139_4.15.0-139.143_all.deb ... 2021-03-24T20:36:43.419 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-headers-4.15.0-139 (4.15.0-139.143) ... 2021-03-24T20:36:48.136 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package linux-headers-4.15.0-139-generic. 2021-03-24T20:36:48.159 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../6-linux-headers-4.15.0-139-generic_4.15.0-139.143_amd64.deb ... 2021-03-24T20:36:48.203 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-headers-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:36:50.782 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../7-linux-headers-generic_4.15.0.139.126_amd64.deb ... 2021-03-24T20:36:50.889 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking linux-headers-generic (4.15.0.139.126) over (4.15.0.136.123) ... 2021-03-24T20:36:51.200 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-modules-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:36:51.777 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-headers-4.15.0-139 (4.15.0-139.143) ... 2021-03-24T20:36:51.903 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-headers-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:36:52.028 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-image-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:36:52.473 INFO:teuthology.orchestra.run.smithi063.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-4.15.0-139-generic 2021-03-24T20:36:52.473 INFO:teuthology.orchestra.run.smithi063.stdout:I: /initrd.img is now a symlink to boot/initrd.img-4.15.0-139-generic 2021-03-24T20:36:52.613 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-modules-extra-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:36:53.090 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-headers-generic (4.15.0.139.126) ... 2021-03-24T20:36:53.207 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-image-generic (4.15.0.139.126) ... 2021-03-24T20:36:53.334 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up linux-generic (4.15.0.139.126) ... 2021-03-24T20:36:53.460 INFO:teuthology.orchestra.run.smithi063.stdout:Processing triggers for linux-image-4.15.0-139-generic (4.15.0-139.143) ... 2021-03-24T20:36:53.687 INFO:teuthology.orchestra.run.smithi063.stdout:/etc/kernel/postinst.d/initramfs-tools: 2021-03-24T20:36:53.687 INFO:teuthology.orchestra.run.smithi063.stdout:update-initramfs: Generating /boot/initrd.img-4.15.0-139-generic 2021-03-24T20:37:09.368 INFO:teuthology.orchestra.run.smithi063.stdout:/etc/kernel/postinst.d/zz-update-grub: 2021-03-24T20:37:09.369 INFO:teuthology.orchestra.run.smithi063.stdout:Sourcing file `/etc/default/grub' 2021-03-24T20:37:09.397 INFO:teuthology.orchestra.run.smithi063.stdout:Generating grub configuration file ... 2021-03-24T20:37:09.781 INFO:teuthology.orchestra.run.smithi063.stdout:Found linux image: /boot/vmlinuz-4.15.0-139-generic 2021-03-24T20:37:09.817 INFO:teuthology.orchestra.run.smithi063.stdout:Found initrd image: /boot/initrd.img-4.15.0-139-generic 2021-03-24T20:37:10.175 INFO:teuthology.orchestra.run.smithi063.stdout:Found linux image: /boot/vmlinuz-4.15.0-136-generic 2021-03-24T20:37:10.202 INFO:teuthology.orchestra.run.smithi063.stdout:Found initrd image: /boot/initrd.img-4.15.0-136-generic 2021-03-24T20:37:11.731 INFO:teuthology.orchestra.run.smithi063.stdout:done 2021-03-24T20:37:12.126 DEBUG:teuthology.orchestra.run.smithi063:> dpkg -s linux-image-generic 2021-03-24T20:37:12.150 INFO:teuthology.orchestra.run.smithi063.stdout:Package: linux-image-generic 2021-03-24T20:37:12.150 INFO:teuthology.orchestra.run.smithi063.stdout:Status: install ok installed 2021-03-24T20:37:12.150 INFO:teuthology.orchestra.run.smithi063.stdout:Priority: optional 2021-03-24T20:37:12.150 INFO:teuthology.orchestra.run.smithi063.stdout:Section: kernel 2021-03-24T20:37:12.151 INFO:teuthology.orchestra.run.smithi063.stdout:Installed-Size: 17 2021-03-24T20:37:12.151 INFO:teuthology.orchestra.run.smithi063.stdout:Maintainer: Ubuntu Kernel Team 2021-03-24T20:37:12.151 INFO:teuthology.orchestra.run.smithi063.stdout:Architecture: amd64 2021-03-24T20:37:12.151 INFO:teuthology.orchestra.run.smithi063.stdout:Source: linux-meta 2021-03-24T20:37:12.152 INFO:teuthology.orchestra.run.smithi063.stdout:Version: 4.15.0.139.126 2021-03-24T20:37:12.152 INFO:teuthology.orchestra.run.smithi063.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-03-24T20:37:12.152 INFO:teuthology.orchestra.run.smithi063.stdout:Depends: linux-image-4.15.0-139-generic, linux-modules-extra-4.15.0-139-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-24T20:37:12.152 INFO:teuthology.orchestra.run.smithi063.stdout:Recommends: thermald 2021-03-24T20:37:12.153 INFO:teuthology.orchestra.run.smithi063.stdout:Description: Generic Linux kernel image 2021-03-24T20:37:12.153 INFO:teuthology.orchestra.run.smithi063.stdout: This package will always depend on the latest generic kernel image 2021-03-24T20:37:12.153 INFO:teuthology.orchestra.run.smithi063.stdout: available. 2021-03-24T20:37:12.153 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.15.0-136-generic Expected: 4.15.0-139-generic 2021-03-24T20:37:12.154 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-03-24T20:37:12.154 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-03-24T20:37:12.154 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-03-24T20:37:12.155 INFO:teuthology.task.kernel:Installing distro kernel on c1.mon.a... 2021-03-24T20:37:12.155 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi052.front.sepia.ceph.com, path=None, version=distro) 2021-03-24T20:37:12.156 DEBUG:teuthology.orchestra.run.smithi052:> sudo apt-get clean 2021-03-24T20:37:12.238 DEBUG:teuthology.orchestra.run.smithi052:> sudo apt-get update 2021-03-24T20:37:12.362 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-03-24T20:37:12.478 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-24T20:37:12.557 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-03-24T20:37:12.645 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-03-24T20:37:14.529 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2021-03-24T20:37:14.578 DEBUG:teuthology.orchestra.run.smithi052:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-24T20:37:14.655 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2021-03-24T20:37:14.956 INFO:teuthology.orchestra.run.smithi052.stdout:Building dependency tree... 2021-03-24T20:37:14.957 INFO:teuthology.orchestra.run.smithi052.stdout:Reading state information... 2021-03-24T20:37:15.051 INFO:teuthology.orchestra.run.smithi052.stderr:E: Unable to locate package linux-image-current-generic 2021-03-24T20:37:15.052 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-24T20:37:15.053 DEBUG:teuthology.orchestra.run.smithi052:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-24T20:37:15.134 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2021-03-24T20:37:15.374 INFO:teuthology.orchestra.run.smithi052.stdout:Building dependency tree... 2021-03-24T20:37:15.375 INFO:teuthology.orchestra.run.smithi052.stdout:Reading state information... 2021-03-24T20:37:15.588 INFO:teuthology.orchestra.run.smithi052.stdout:linux-image-generic is already the newest version (4.15.0.139.126). 2021-03-24T20:37:15.589 INFO:teuthology.orchestra.run.smithi052.stdout:0 upgraded, 0 newly installed, 0 to remove and 30 not upgraded. 2021-03-24T20:37:15.590 DEBUG:teuthology.orchestra.run.smithi052:> dpkg -s linux-image-generic 2021-03-24T20:37:15.614 INFO:teuthology.orchestra.run.smithi052.stdout:Package: linux-image-generic 2021-03-24T20:37:15.615 INFO:teuthology.orchestra.run.smithi052.stdout:Status: install ok installed 2021-03-24T20:37:15.615 INFO:teuthology.orchestra.run.smithi052.stdout:Priority: optional 2021-03-24T20:37:15.615 INFO:teuthology.orchestra.run.smithi052.stdout:Section: kernel 2021-03-24T20:37:15.615 INFO:teuthology.orchestra.run.smithi052.stdout:Installed-Size: 17 2021-03-24T20:37:15.615 INFO:teuthology.orchestra.run.smithi052.stdout:Maintainer: Ubuntu Kernel Team 2021-03-24T20:37:15.616 INFO:teuthology.orchestra.run.smithi052.stdout:Architecture: amd64 2021-03-24T20:37:15.616 INFO:teuthology.orchestra.run.smithi052.stdout:Source: linux-meta 2021-03-24T20:37:15.616 INFO:teuthology.orchestra.run.smithi052.stdout:Version: 4.15.0.139.126 2021-03-24T20:37:15.617 INFO:teuthology.orchestra.run.smithi052.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-03-24T20:37:15.617 INFO:teuthology.orchestra.run.smithi052.stdout:Depends: linux-image-4.15.0-139-generic, linux-modules-extra-4.15.0-139-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-24T20:37:15.617 INFO:teuthology.orchestra.run.smithi052.stdout:Recommends: thermald 2021-03-24T20:37:15.617 INFO:teuthology.orchestra.run.smithi052.stdout:Description: Generic Linux kernel image 2021-03-24T20:37:15.618 INFO:teuthology.orchestra.run.smithi052.stdout: This package will always depend on the latest generic kernel image 2021-03-24T20:37:15.618 INFO:teuthology.orchestra.run.smithi052.stdout: available. 2021-03-24T20:37:15.618 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2021-03-24T20:37:15.662 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.kKl3fx0DD8 2021-03-24T20:37:15.662 DEBUG:teuthology.orchestra.run.smithi052:> sudo cp /boot/grub/grub.cfg /tmp/tmp.kKl3fx0DD8 2021-03-24T20:37:15.718 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.kKl3fx0DD8 2021-03-24T20:37:15.830 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.kKl3fx0DD8 is 8KB 2021-03-24T20:37:15.882 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.kKl3fx0DD8 2021-03-24T20:37:15.889 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-03-24T20:37:15.947 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:37:15.947 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-03-24T20:37:15.948 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-03-24T20:37:16.018 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-139-generic 2021-03-24T20:37:16.019 DEBUG:teuthology.orchestra.run.smithi052:> sudo update-grub 2021-03-24T20:37:17.381 INFO:teuthology.orchestra.run.smithi052.stderr:Sourcing file `/etc/default/grub' 2021-03-24T20:37:17.392 INFO:teuthology.orchestra.run.smithi052.stderr:Generating grub configuration file ... 2021-03-24T20:37:17.685 INFO:teuthology.orchestra.run.smithi052.stderr:Found linux image: /boot/vmlinuz-4.15.0-139-generic 2021-03-24T20:37:17.695 INFO:teuthology.orchestra.run.smithi052.stderr:Found initrd image: /boot/initrd.img-4.15.0-139-generic 2021-03-24T20:37:18.081 INFO:teuthology.orchestra.run.smithi052.stderr:Found linux image: /boot/vmlinuz-4.15.0-136-generic 2021-03-24T20:37:18.095 INFO:teuthology.orchestra.run.smithi052.stderr:Found initrd image: /boot/initrd.img-4.15.0-136-generic 2021-03-24T20:37:19.208 INFO:teuthology.orchestra.run.smithi052.stderr:done 2021-03-24T20:37:19.210 DEBUG:teuthology.orchestra.run.smithi052:> sudo shutdown -r now 2021-03-24T20:37:19.256 INFO:teuthology.task.kernel:Installing distro kernel on c2.mon.a... 2021-03-24T20:37:19.256 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi063.front.sepia.ceph.com, path=None, version=distro) 2021-03-24T20:37:19.257 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-get clean 2021-03-24T20:37:19.324 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-get update 2021-03-24T20:37:19.577 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-03-24T20:37:19.577 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-24T20:37:19.645 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:3 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-03-24T20:37:19.725 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-backports InRelease 2021-03-24T20:37:21.525 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2021-03-24T20:37:21.575 DEBUG:teuthology.orchestra.run.smithi063:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-current-generic 2021-03-24T20:37:21.657 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2021-03-24T20:37:21.900 INFO:teuthology.orchestra.run.smithi063.stdout:Building dependency tree... 2021-03-24T20:37:21.900 INFO:teuthology.orchestra.run.smithi063.stdout:Reading state information... 2021-03-24T20:37:21.988 INFO:teuthology.orchestra.run.smithi063.stderr:E: Unable to locate package linux-image-current-generic 2021-03-24T20:37:21.989 DEBUG:teuthology.orchestra.run:got remote process result: 100 2021-03-24T20:37:21.990 DEBUG:teuthology.orchestra.run.smithi063:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2021-03-24T20:37:22.069 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2021-03-24T20:37:22.346 INFO:teuthology.orchestra.run.smithi063.stdout:Building dependency tree... 2021-03-24T20:37:22.346 INFO:teuthology.orchestra.run.smithi063.stdout:Reading state information... 2021-03-24T20:37:22.568 INFO:teuthology.orchestra.run.smithi063.stdout:linux-image-generic is already the newest version (4.15.0.139.126). 2021-03-24T20:37:22.568 INFO:teuthology.orchestra.run.smithi063.stdout:0 upgraded, 0 newly installed, 0 to remove and 30 not upgraded. 2021-03-24T20:37:22.570 DEBUG:teuthology.orchestra.run.smithi063:> dpkg -s linux-image-generic 2021-03-24T20:37:22.595 INFO:teuthology.orchestra.run.smithi063.stdout:Package: linux-image-generic 2021-03-24T20:37:22.596 INFO:teuthology.orchestra.run.smithi063.stdout:Status: install ok installed 2021-03-24T20:37:22.596 INFO:teuthology.orchestra.run.smithi063.stdout:Priority: optional 2021-03-24T20:37:22.596 INFO:teuthology.orchestra.run.smithi063.stdout:Section: kernel 2021-03-24T20:37:22.596 INFO:teuthology.orchestra.run.smithi063.stdout:Installed-Size: 17 2021-03-24T20:37:22.597 INFO:teuthology.orchestra.run.smithi063.stdout:Maintainer: Ubuntu Kernel Team 2021-03-24T20:37:22.597 INFO:teuthology.orchestra.run.smithi063.stdout:Architecture: amd64 2021-03-24T20:37:22.597 INFO:teuthology.orchestra.run.smithi063.stdout:Source: linux-meta 2021-03-24T20:37:22.597 INFO:teuthology.orchestra.run.smithi063.stdout:Version: 4.15.0.139.126 2021-03-24T20:37:22.598 INFO:teuthology.orchestra.run.smithi063.stdout:Provides: wireguard-modules (= 1.0.20201112-1~18.04.1), zfs-modules (= 0.7.5-1ubuntu16.10) 2021-03-24T20:37:22.598 INFO:teuthology.orchestra.run.smithi063.stdout:Depends: linux-image-4.15.0-139-generic, linux-modules-extra-4.15.0-139-generic, linux-firmware, intel-microcode, amd64-microcode 2021-03-24T20:37:22.598 INFO:teuthology.orchestra.run.smithi063.stdout:Recommends: thermald 2021-03-24T20:37:22.598 INFO:teuthology.orchestra.run.smithi063.stdout:Description: Generic Linux kernel image 2021-03-24T20:37:22.599 INFO:teuthology.orchestra.run.smithi063.stdout: This package will always depend on the latest generic kernel image 2021-03-24T20:37:22.599 INFO:teuthology.orchestra.run.smithi063.stdout: available. 2021-03-24T20:37:22.599 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2021-03-24T20:37:22.641 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.QeHX4aOhE7 2021-03-24T20:37:22.642 DEBUG:teuthology.orchestra.run.smithi063:> sudo cp /boot/grub/grub.cfg /tmp/tmp.QeHX4aOhE7 2021-03-24T20:37:22.710 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.QeHX4aOhE7 2021-03-24T20:37:22.822 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.QeHX4aOhE7 is 8KB 2021-03-24T20:37:22.874 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.QeHX4aOhE7 2021-03-24T20:37:22.882 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2021-03-24T20:37:22.937 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:37:22.938 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/grub.d/01_ceph_kernel 2021-03-24T20:37:22.939 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2021-03-24T20:37:23.017 INFO:teuthology.task.kernel:Distro Kernel Version: 4.15.0-139-generic 2021-03-24T20:37:23.018 DEBUG:teuthology.orchestra.run.smithi063:> sudo update-grub 2021-03-24T20:37:24.305 INFO:teuthology.orchestra.run.smithi063.stderr:Sourcing file `/etc/default/grub' 2021-03-24T20:37:24.330 INFO:teuthology.orchestra.run.smithi063.stderr:Generating grub configuration file ... 2021-03-24T20:37:24.617 INFO:teuthology.orchestra.run.smithi063.stderr:Found linux image: /boot/vmlinuz-4.15.0-139-generic 2021-03-24T20:37:24.682 INFO:teuthology.orchestra.run.smithi063.stderr:Found initrd image: /boot/initrd.img-4.15.0-139-generic 2021-03-24T20:37:24.966 INFO:teuthology.orchestra.run.smithi063.stderr:Found linux image: /boot/vmlinuz-4.15.0-136-generic 2021-03-24T20:37:24.979 INFO:teuthology.orchestra.run.smithi063.stderr:Found initrd image: /boot/initrd.img-4.15.0-136-generic 2021-03-24T20:37:26.170 INFO:teuthology.orchestra.run.smithi063.stderr:done 2021-03-24T20:37:26.172 DEBUG:teuthology.orchestra.run.smithi063:> sudo shutdown -r now 2021-03-24T20:37:56.202 INFO:teuthology.misc:Re-opening connections... 2021-03-24T20:37:56.203 INFO:teuthology.misc:trying to connect to ubuntu@smithi052.front.sepia.ceph.com 2021-03-24T20:37:56.204 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-24T20:37:56.205 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T20:38:56.263 DEBUG:teuthology.orchestra.remote:timed out 2021-03-24T20:38:56.299 INFO:teuthology.misc:trying to connect to ubuntu@smithi063.front.sepia.ceph.com 2021-03-24T20:38:56.300 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-24T20:38:56.301 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T20:38:59.362 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2021-03-24T20:38:59.363 DEBUG:teuthology.misc:waited 63.16000413894653 2021-03-24T20:39:00.365 INFO:teuthology.misc:trying to connect to ubuntu@smithi052.front.sepia.ceph.com 2021-03-24T20:39:00.365 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-24T20:39:00.366 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T20:39:00.450 DEBUG:teuthology.orchestra.run.smithi052:> true 2021-03-24T20:39:00.783 DEBUG:teuthology.misc:waited 64.5802161693573 2021-03-24T20:39:01.784 INFO:teuthology.misc:trying to connect to ubuntu@smithi063.front.sepia.ceph.com 2021-03-24T20:39:01.785 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-24T20:39:01.786 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T20:39:02.434 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2021-03-24T20:39:02.435 DEBUG:teuthology.misc:waited 66.2317545413971 2021-03-24T20:39:03.436 INFO:teuthology.misc:trying to connect to ubuntu@smithi063.front.sepia.ceph.com 2021-03-24T20:39:03.436 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-24T20:39:03.437 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T20:39:06.498 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.63 2021-03-24T20:39:06.499 DEBUG:teuthology.misc:waited 70.29583048820496 2021-03-24T20:39:07.500 INFO:teuthology.misc:trying to connect to ubuntu@smithi063.front.sepia.ceph.com 2021-03-24T20:39:07.501 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-24T20:39:07.502 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T20:39:07.694 DEBUG:teuthology.orchestra.remote:No existing session 2021-03-24T20:39:07.699 DEBUG:teuthology.misc:waited 71.49553680419922 2021-03-24T20:39:08.701 INFO:teuthology.misc:trying to connect to ubuntu@smithi063.front.sepia.ceph.com 2021-03-24T20:39:08.702 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-24T20:39:08.702 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T20:39:08.765 DEBUG:teuthology.orchestra.run.smithi063:> true 2021-03-24T20:39:09.310 DEBUG:teuthology.misc:waited 73.10671186447144 2021-03-24T20:39:10.311 INFO:teuthology.task.kernel:Checking client c1.mon.a for new kernel version... 2021-03-24T20:39:10.312 INFO:teuthology.task.kernel:Checking kernel version of c1.mon.a, want "4.15.0-139-generic"... 2021-03-24T20:39:10.312 DEBUG:teuthology.orchestra.run.smithi052:> uname -r 2021-03-24T20:39:10.319 INFO:teuthology.orchestra.run.smithi052.stdout:4.15.0-139-generic 2021-03-24T20:39:10.319 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-139-generic vs 4.15.0-139-generic 2021-03-24T20:39:10.320 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-03-24T20:39:10.320 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-03-24T20:39:10.320 INFO:teuthology.task.kernel:Checking client c2.mon.a for new kernel version... 2021-03-24T20:39:10.321 INFO:teuthology.task.kernel:Checking kernel version of c2.mon.a, want "4.15.0-139-generic"... 2021-03-24T20:39:10.321 DEBUG:teuthology.orchestra.run.smithi063:> uname -r 2021-03-24T20:39:10.328 INFO:teuthology.orchestra.run.smithi063.stdout:4.15.0-139-generic 2021-03-24T20:39:10.328 DEBUG:teuthology.task.kernel:current kernel version is 4.15.0-139-generic vs 4.15.0-139-generic 2021-03-24T20:39:10.328 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-03-24T20:39:10.329 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2021-03-24T20:39:11.330 INFO:teuthology.task.kernel:Enabling kdb on c1.mon.a... 2021-03-24T20:39:11.331 DEBUG:teuthology.orchestra.run.smithi052:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-24T20:39:11.356 INFO:teuthology.orchestra.run.smithi052.stdout:ttyS1 2021-03-24T20:39:11.377 INFO:teuthology.task.kernel:Enabling kdb on c2.mon.a... 2021-03-24T20:39:11.377 DEBUG:teuthology.orchestra.run.smithi063:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-03-24T20:39:11.414 INFO:teuthology.orchestra.run.smithi063.stdout:ttyS1 2021-03-24T20:39:11.461 INFO:teuthology.run_tasks:Running task internal.base... 2021-03-24T20:39:11.502 INFO:teuthology.task.internal:Creating test directory... 2021-03-24T20:39:11.504 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-24T20:39:11.507 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-03-24T20:39:11.514 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-03-24T20:39:11.631 INFO:teuthology.run_tasks:Running task internal.archive... 2021-03-24T20:39:11.640 INFO:teuthology.task.internal:Creating archive directory... 2021-03-24T20:39:11.640 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-24T20:39:11.644 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-03-24T20:39:11.692 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-03-24T20:39:11.701 INFO:teuthology.task.internal:Enabling coredump saving... 2021-03-24T20:39:11.701 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-24T20:39:11.704 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-03-24T20:39:11.728 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-24T20:39:11.738 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-24T20:39:11.745 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-24T20:39:11.751 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-03-24T20:39:11.754 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-03-24T20:39:11.769 INFO:teuthology.task.internal:Configuring sudo... 2021-03-24T20:39:11.770 DEBUG:teuthology.orchestra.run.smithi052:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-24T20:39:11.784 DEBUG:teuthology.orchestra.run.smithi063:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-03-24T20:39:11.810 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-03-24T20:39:11.821 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-03-24T20:39:11.878 DEBUG:teuthology.orchestra.run.smithi052:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-24T20:39:11.881 DEBUG:teuthology.orchestra.run.smithi063:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-03-24T20:39:11.887 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-24T20:39:11.932 DEBUG:teuthology.orchestra.run.smithi052:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-24T20:39:11.980 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:39:11.986 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-24T20:39:12.041 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-03-24T20:39:12.048 DEBUG:teuthology.orchestra.run.smithi063:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-03-24T20:39:12.098 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:39:12.098 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-03-24T20:39:12.159 DEBUG:teuthology.orchestra.run.smithi052:> sudo service rsyslog restart 2021-03-24T20:39:12.189 DEBUG:teuthology.orchestra.run.smithi063:> sudo service rsyslog restart 2021-03-24T20:39:12.247 INFO:teuthology.run_tasks:Running task internal.timer... 2021-03-24T20:39:12.258 INFO:teuthology.task.internal:Starting timer... 2021-03-24T20:39:12.258 INFO:teuthology.run_tasks:Running task pcp... 2021-03-24T20:39:12.333 INFO:teuthology.run_tasks:Running task selinux... 2021-03-24T20:39:12.442 DEBUG:teuthology.task.selinux:Excluding smithi052: OS 'ubuntu' does not support SELinux 2021-03-24T20:39:12.443 DEBUG:teuthology.task.selinux:Excluding smithi063: OS 'ubuntu' does not support SELinux 2021-03-24T20:39:12.443 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-03-24T20:39:12.443 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2021-03-24T20:39:12.444 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-03-24T20:39:12.444 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-03-24T20:39:12.463 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-03-24T20:39:12.464 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-03-24T20:39:12.565 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-03-24T20:39:12.572 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi052.front.sepia.ceph.com,smithi063.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-03-24T20:39:13.459 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-03-24T20:39:15.306 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-03-24T20:39:17.789 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-03-24T20:39:17.833 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:17 +0000 (0:00:00.079) 0:00:00.079 ******* 2021-03-24T20:39:17.845 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:17 +0000 (0:00:00.012) 0:00:00.091 ******* 2021-03-24T20:39:19.584 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* 2021-03-24T20:39:19.592 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi052.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi052.front.sepia.ceph.com] 2021-03-24T20:39:19.641 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:19 +0000 (0:00:01.795) 0:00:01.886 ******* 2021-03-24T20:39:19.656 INFO:teuthology.task.ansible.out:[DEPRECATION WARNING]: Distribution Ubuntu 18.04 on host smithi063.front.sepia.ceph.com should use /usr/bin/python3, but is using /usr/bin/python for backward compatibility with prior Ansible releases. A future Ansible release will default to using the discovered platform python for this host. See https://docs.ansible.com/ansible/2.9/reference_appendices/inter preter_discovery.html for more information. This feature will be removed in version 2.12. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg. ok: [smithi063.front.sepia.ceph.com] 2021-03-24T20:39:19.711 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:19 +0000 (0:00:00.070) 0:00:01.956 ******* 2021-03-24T20:39:19.915 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-03-24T20:39:19.917 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-24T20:39:19.939 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:19 +0000 (0:00:00.228) 0:00:02.185 ******* 2021-03-24T20:39:19.967 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:19 +0000 (0:00:00.027) 0:00:02.212 ******* 2021-03-24T20:39:20.593 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-03-24T20:39:20.798 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} Wednesday 24 March 2021 20:39:20 +0000 (0:00:00.653) 0:00:02.865 ******* changed: [smithi063.front.sepia.ceph.com] => {"append": false, "changed": true, "comment": "cm,,,", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} Wednesday 24 March 2021 20:39:20 +0000 (0:00:00.034) 0:00:02.900 ******* 2021-03-24T20:39:20.996 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-03-24T20:39:20.998 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.069212", "end": "2021-03-24 20:39:20.521466", "rc": 0, "start": "2021-03-24 20:39:20.452254", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["passwd", "-d", "cm"], "delta": "0:00:00.072667", "end": "2021-03-24 20:39:20.748131", "rc": 0, "start": "2021-03-24 20:39:20.675464", "stderr": "", "stderr_lines": [], "stdout": "passwd: password expiry information changed.", "stdout_lines": ["passwd: password expiry information changed."]} 2021-03-24T20:39:21.021 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:21 +0000 (0:00:00.366) 0:00:03.266 ******* 2021-03-24T20:39:21.042 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:21 +0000 (0:00:00.020) 0:00:03.287 ******* 2021-03-24T20:39:21.303 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-03-24T20:39:21.363 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} Wednesday 24 March 2021 20:39:21 +0000 (0:00:00.288) 0:00:03.576 ******* Wednesday 24 March 2021 20:39:21 +0000 (0:00:00.021) 0:00:03.597 ******* 2021-03-24T20:39:22.261 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616618361.36811-17101-128639530169572/source", "state": "file", "uid": 0} 2021-03-24T20:39:22.284 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:22 +0000 (0:00:00.932) 0:00:04.530 ******* 2021-03-24T20:39:22.292 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616618361.3889396-17105-126584215972356/source", "state": "file", "uid": 0} 2021-03-24T20:39:22.332 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:22 +0000 (0:00:00.048) 0:00:04.578 ******* 2021-03-24T20:39:22.814 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-03-24T20:39:22.816 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-03-24T20:39:22.818 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-24T20:39:22.838 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-03-24T20:39:22.840 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-03-24T20:39:22.919 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-03-24T20:39:22.923 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-03-24T20:39:23.107 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:23 +0000 (0:00:00.774) 0:00:05.352 ******* 2021-03-24T20:39:23.145 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:23 +0000 (0:00:00.020) 0:00:05.373 ******* 2021-03-24T20:39:23.870 INFO:teuthology.task.ansible.out: 2021-03-24T20:39:23.872 INFO:teuthology.task.ansible.out:TASK [Gathering Facts] ********************************************************* ok: [smithi063.front.sepia.ceph.com] 2021-03-24T20:39:23.881 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] 2021-03-24T20:39:23.946 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. Wednesday 24 March 2021 20:39:23 +0000 (0:00:00.819) 0:00:06.192 ******* 2021-03-24T20:39:23.974 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:23 +0000 (0:00:00.027) 0:00:06.220 ******* 2021-03-24T20:39:24.424 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-03-24T20:39:24.426 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.186049", "end": "2021-03-24 20:39:24.392703", "rc": 0, "start": "2021-03-24 20:39:24.206654", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} 2021-03-24T20:39:24.448 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:24 +0000 (0:00:00.474) 0:00:06.694 ******* 2021-03-24T20:39:24.522 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.187355", "end": "2021-03-24 20:39:24.496401", "rc": 0, "start": "2021-03-24 20:39:24.309046", "stderr": "", "stderr_lines": [], "stdout": "c31004c4720a55ac12a95d66c32e64f592877604", "stdout_lines": ["c31004c4720a55ac12a95d66c32e64f592877604"]} 2021-03-24T20:39:24.549 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:24 +0000 (0:00:00.100) 0:00:06.795 ******* 2021-03-24T20:39:24.601 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-03-24T20:39:24.604 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.014628", "end": "2021-03-24 20:39:24.353814", "failed_when_result": false, "rc": 0, "start": "2021-03-24 20:39:24.339186", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-24T20:39:24.634 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:24 +0000 (0:00:00.084) 0:00:06.879 ******* 2021-03-24T20:39:24.706 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-03-24T20:39:24.708 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:39:24.730 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:24 +0000 (0:00:00.096) 0:00:06.976 ******* 2021-03-24T20:39:24.738 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-03-24T20:39:24.740 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.013229", "end": "2021-03-24 20:39:24.268408", "failed_when_result": false, "rc": 0, "start": "2021-03-24 20:39:24.255179", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-03-24T20:39:24.768 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:24 +0000 (0:00:00.037) 0:00:07.014 ******* 2021-03-24T20:39:24.807 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ 2021-03-24T20:39:24.809 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:39:24.831 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:24 +0000 (0:00:00.062) 0:00:07.077 ******* 2021-03-24T20:39:24.848 INFO:teuthology.task.ansible.out: TASK [Determine if we can skip users and pubkeys updates] ********************** 2021-03-24T20:39:24.850 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:39:24.874 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:24 +0000 (0:00:00.043) 0:00:07.120 ******* 2021-03-24T20:39:24.912 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-03-24T20:39:24.914 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:39:24.940 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:24 +0000 (0:00:00.061) 0:00:07.182 ******* 2021-03-24T20:39:24.954 INFO:teuthology.task.ansible.out: TASK [Merge extra_admin_users into managed_admin_users] ************************ skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:39:24.979 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:24 +0000 (0:00:00.043) 0:00:07.225 ******* 2021-03-24T20:39:25.009 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-03-24T20:39:25.010 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:39:25.039 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:25 +0000 (0:00:00.058) 0:00:07.285 ******* 2021-03-24T20:39:25.057 INFO:teuthology.task.ansible.out: TASK [Remove managed_admin_users from managed_users] *************************** 2021-03-24T20:39:25.059 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:39:25.083 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:25 +0000 (0:00:00.044) 0:00:07.329 ******* 2021-03-24T20:39:25.111 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-03-24T20:39:25.113 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:39:25.136 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:25 +0000 (0:00:00.052) 0:00:07.382 ******* 2021-03-24T20:39:25.155 INFO:teuthology.task.ansible.out: TASK [Filter the managed_users list] ******************************************* 2021-03-24T20:39:25.156 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:39:25.180 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:25 +0000 (0:00:00.043) 0:00:07.425 ******* 2021-03-24T20:39:25.249 INFO:teuthology.task.ansible.out: TASK [Filter the managed_admin_users list] ************************************* 2021-03-24T20:39:25.251 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:39:25.273 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:25 +0000 (0:00:00.093) 0:00:07.519 ******* 2021-03-24T20:39:25.322 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T20:39:25.346 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:25 +0000 (0:00:00.072) 0:00:07.592 ******* 2021-03-24T20:39:25.449 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T20:39:25.475 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:25 +0000 (0:00:00.129) 0:00:07.721 ******* 2021-03-24T20:39:25.609 INFO:teuthology.task.ansible.out: TASK [Create all admin users with sudo access.] ******************************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-03-24T20:39:25.735 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-03-24T20:39:25.757 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-24T20:39:25.884 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-03-24T20:39:25.906 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-24T20:39:26.023 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-03-24T20:39:26.045 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-24T20:39:26.174 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-03-24T20:39:26.176 INFO:teuthology.task.ansible.out: 2021-03-24T20:39:26.199 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-24T20:39:26.326 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-03-24T20:39:26.339 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-24T20:39:26.476 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-03-24T20:39:26.498 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-24T20:39:26.499 INFO:teuthology.task.ansible.out: 2021-03-24T20:39:26.623 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-03-24T20:39:26.645 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-24T20:39:26.767 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-03-24T20:39:26.811 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-03-24T20:39:26.918 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-03-24T20:39:26.964 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-24T20:39:27.072 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-03-24T20:39:27.105 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-24T20:39:27.215 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-03-24T20:39:27.258 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-24T20:39:27.368 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-03-24T20:39:27.423 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-24T20:39:27.509 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-03-24T20:39:27.562 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-24T20:39:27.661 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-03-24T20:39:27.718 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-24T20:39:27.821 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-03-24T20:39:27.865 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-24T20:39:27.974 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-03-24T20:39:27.976 INFO:teuthology.task.ansible.out: 2021-03-24T20:39:28.009 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-24T20:39:28.127 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-03-24T20:39:28.150 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-24T20:39:28.266 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-03-24T20:39:28.289 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-24T20:39:28.416 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-03-24T20:39:28.438 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-24T20:39:28.565 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-03-24T20:39:28.567 INFO:teuthology.task.ansible.out: 2021-03-24T20:39:28.588 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-24T20:39:28.717 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-03-24T20:39:28.730 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-03-24T20:39:28.861 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1122, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1122} 2021-03-24T20:39:28.882 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-24T20:39:28.999 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-03-24T20:39:29.042 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-24T20:39:29.148 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-03-24T20:39:29.193 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-24T20:39:29.300 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-03-24T20:39:29.334 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-24T20:39:29.450 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-03-24T20:39:29.482 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-24T20:39:29.603 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-03-24T20:39:29.649 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-24T20:39:29.757 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-03-24T20:39:29.800 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-24T20:39:29.907 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-03-24T20:39:29.962 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-24T20:39:30.049 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-03-24T20:39:30.113 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-24T20:39:30.199 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-03-24T20:39:30.262 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-24T20:39:30.338 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-03-24T20:39:30.404 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-24T20:39:30.489 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-03-24T20:39:30.552 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-24T20:39:30.638 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-03-24T20:39:30.693 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-24T20:39:30.778 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-03-24T20:39:30.841 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-24T20:39:30.918 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-03-24T20:39:30.993 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-03-24T20:39:31.139 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1137} ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-24T20:39:31.208 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-03-24T20:39:31.287 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-24T20:39:31.352 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-24T20:39:31.437 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-24T20:39:31.502 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-03-24T20:39:31.587 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-24T20:39:31.650 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-03-24T20:39:31.737 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-24T20:39:31.790 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-03-24T20:39:31.886 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-24T20:39:31.940 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-03-24T20:39:32.036 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-24T20:39:32.079 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-03-24T20:39:32.186 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-24T20:39:32.224 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-03-24T20:39:32.341 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-24T20:39:32.375 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-03-24T20:39:32.502 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-24T20:39:32.524 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-24T20:39:32.652 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-24T20:39:32.667 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-03-24T20:39:32.804 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-03-24T20:39:32.850 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-03-24T20:39:32.954 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-24T20:39:32.966 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-03-24T20:39:33.106 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-24T20:39:33.108 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-03-24T20:39:33.249 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-24T20:39:33.272 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-03-24T20:39:33.398 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-24T20:39:33.430 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-03-24T20:39:33.558 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-24T20:39:33.570 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-03-24T20:39:33.697 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-03-24T20:39:33.731 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-03-24T20:39:33.862 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-03-24T20:39:33.873 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-03-24T20:39:34.010 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-03-24T20:39:34.022 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-03-24T20:39:34.151 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-24T20:39:34.173 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-03-24T20:39:34.300 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-24T20:39:34.312 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-03-24T20:39:34.449 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-24T20:39:34.460 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-03-24T20:39:34.600 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-24T20:39:34.617 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-03-24T20:39:34.889 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-24T20:39:35.028 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-03-24T20:39:35.096 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-24T20:39:35.172 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-03-24T20:39:35.258 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-24T20:39:35.332 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-03-24T20:39:35.408 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-24T20:39:35.471 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-03-24T20:39:35.555 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-24T20:39:35.631 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-03-24T20:39:35.716 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-24T20:39:35.781 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-03-24T20:39:35.866 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-24T20:39:35.920 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-03-24T20:39:36.028 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-24T20:39:36.072 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-03-24T20:39:36.170 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-24T20:39:36.241 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-03-24T20:39:36.412 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-24T20:39:36.476 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-03-24T20:39:36.604 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-24T20:39:36.648 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-03-24T20:39:36.755 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-24T20:39:36.799 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-03-24T20:39:36.801 INFO:teuthology.task.ansible.out: 2021-03-24T20:39:36.907 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-24T20:39:36.950 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-03-24T20:39:37.068 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-24T20:39:37.101 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-03-24T20:39:37.226 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-24T20:39:37.248 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-03-24T20:39:37.376 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-24T20:39:37.398 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-03-24T20:39:37.528 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-24T20:39:37.549 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-03-24T20:39:37.680 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-24T20:39:37.692 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-03-24T20:39:37.824 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-24T20:39:37.836 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-03-24T20:39:37.975 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-24T20:39:37.986 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-03-24T20:39:38.124 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-24T20:39:38.135 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-03-24T20:39:38.284 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-24T20:39:38.285 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-03-24T20:39:38.436 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-24T20:39:38.437 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-03-24T20:39:38.576 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-24T20:39:38.599 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-03-24T20:39:38.726 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-24T20:39:38.748 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-03-24T20:39:38.877 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-24T20:39:38.899 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-03-24T20:39:39.049 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-24T20:39:39.058 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-03-24T20:39:39.208 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-03-24T20:39:39.365 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-24T20:39:39.370 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-03-24T20:39:39.529 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-24T20:39:39.539 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-03-24T20:39:39.669 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-24T20:39:39.692 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-03-24T20:39:39.829 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-24T20:39:39.852 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-03-24T20:39:39.978 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-24T20:39:40.011 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-03-24T20:39:40.129 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-24T20:39:40.173 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-03-24T20:39:40.291 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-24T20:39:40.336 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-03-24T20:39:40.442 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-24T20:39:40.508 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-03-24T20:39:40.602 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-24T20:39:40.656 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-03-24T20:39:40.763 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-24T20:39:40.808 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-03-24T20:39:40.914 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-24T20:39:40.979 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-03-24T20:39:41.076 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-24T20:39:41.132 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-03-24T20:39:41.227 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-24T20:39:41.281 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-03-24T20:39:41.378 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-24T20:39:41.432 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-03-24T20:39:41.518 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-24T20:39:41.594 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-03-24T20:39:41.672 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-24T20:39:41.738 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-03-24T20:39:41.812 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-24T20:39:41.886 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-03-24T20:39:41.966 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-24T20:39:42.031 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-03-24T20:39:42.118 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-24T20:39:42.183 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-03-24T20:39:42.268 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-24T20:39:42.343 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-03-24T20:39:42.417 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-24T20:39:42.482 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-03-24T20:39:42.591 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-24T20:39:42.636 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-03-24T20:39:42.745 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-24T20:39:42.790 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-03-24T20:39:42.896 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-24T20:39:42.951 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-03-24T20:39:43.060 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-24T20:39:43.115 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-03-24T20:39:43.201 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-24T20:39:43.267 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-03-24T20:39:43.341 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-24T20:39:43.344 INFO:teuthology.task.ansible.out: 2021-03-24T20:39:43.421 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-03-24T20:39:43.497 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-24T20:39:43.582 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-03-24T20:39:43.639 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-24T20:39:43.727 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-03-24T20:39:43.804 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-24T20:39:43.880 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-03-24T20:39:43.968 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-24T20:39:44.033 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-03-24T20:39:44.119 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-24T20:39:44.183 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-03-24T20:39:44.271 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-03-24T20:39:44.335 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-03-24T20:39:44.442 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-24T20:39:44.484 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-03-24T20:39:44.581 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-24T20:39:44.635 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-03-24T20:39:44.730 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-24T20:39:44.784 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-03-24T20:39:44.890 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-24T20:39:45.012 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-03-24T20:39:45.055 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-24T20:39:45.057 INFO:teuthology.task.ansible.out: 2021-03-24T20:39:45.188 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-03-24T20:39:45.210 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-24T20:39:45.368 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-03-24T20:39:45.396 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-24T20:39:45.535 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-03-24T20:39:45.548 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-24T20:39:45.685 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-03-24T20:39:45.708 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-24T20:39:45.833 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-03-24T20:39:45.845 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-24T20:39:45.972 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-03-24T20:39:46.004 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-24T20:39:46.130 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-03-24T20:39:46.152 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-24T20:39:46.279 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-03-24T20:39:46.281 INFO:teuthology.task.ansible.out: 2021-03-24T20:39:46.292 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-24T20:39:46.421 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-03-24T20:39:46.455 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-03-24T20:39:46.572 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1234, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "move_home": false, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "uid": 1234} 2021-03-24T20:39:46.604 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-24T20:39:46.711 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "move_home": false, "name": "adking", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-03-24T20:39:46.758 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-03-24T20:39:46.875 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1236, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "move_home": false, "name": "amathuri", "shell": "/bin/bash", "state": "present", "uid": 1236} 2021-03-24T20:39:46.918 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-03-24T20:39:47.132 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1237, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "move_home": false, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "uid": 1237} 2021-03-24T20:39:47.225 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:47 +0000 (0:00:21.750) 0:00:29.471 ******* 2021-03-24T20:39:47.248 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:47 +0000 (0:00:00.022) 0:00:29.494 ******* 2021-03-24T20:39:47.313 INFO:teuthology.task.ansible.out: TASK [Create all users without sudo access.] *********************************** 2021-03-24T20:39:47.337 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:47 +0000 (0:00:00.088) 0:00:29.583 ******* 2021-03-24T20:39:47.370 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:47 +0000 (0:00:00.033) 0:00:29.616 ******* 2021-03-24T20:39:47.422 INFO:teuthology.task.ansible.out: TASK [Merge managed_users and managed_admin users] ***************************** 2021-03-24T20:39:47.424 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-24T20:39:47.426 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-24T20:39:47.428 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-24T20:39:47.429 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-24T20:39:47.431 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-24T20:39:47.432 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-24T20:39:47.434 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a 2021-03-24T20:39:47.434 INFO:teuthology.task.ansible.out:4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e 2021-03-24T20:39:47.436 INFO:teuthology.task.ansible.out:7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef14 2021-03-24T20:39:47.438 INFO:teuthology.task.ansible.out:80c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d301 2021-03-24T20:39:47.439 INFO:teuthology.task.ansible.out:7b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214 2021-03-24T20:39:47.441 INFO:teuthology.task.ansible.out:bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelope 2021-03-24T20:39:47.442 INFO:teuthology.task.ansible.out:z@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed 2021-03-24T20:39:47.444 INFO:teuthology.task.ansible.out:19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174f 2021-03-24T20:39:47.445 INFO:teuthology.task.ansible.out:e16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-24T20:39:47.473 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:47 +0000 (0:00:00.103) 0:00:29.719 ******* 2021-03-24T20:39:47.486 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-03-24T20:39:47.488 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-03-24T20:39:47.489 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop K 2021-03-24T20:39:47.491 INFO:teuthology.task.ansible.out:D0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\ 2021-03-24T20:39:47.491 INFO:teuthology.task.ansible.out:nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9 2021-03-24T20:39:47.493 INFO:teuthology.task.ansible.out:df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 2021-03-24T20:39:47.494 INFO:teuthology.task.ansible.out: 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc9795 2021-03-24T20:39:47.496 INFO:teuthology.task.ansible.out:3cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e 2021-03-24T20:39:47.498 INFO:teuthology.task.ansible.out:7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef14 2021-03-24T20:39:47.499 INFO:teuthology.task.ansible.out:80c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d301 2021-03-24T20:39:47.501 INFO:teuthology.task.ansible.out:7b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214 2021-03-24T20:39:47.503 INFO:teuthology.task.ansible.out:bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "o 2021-03-24T20:39:47.503 INFO:teuthology.task.ansible.out:vpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelope 2021-03-24T20:39:47.505 INFO:teuthology.task.ansible.out:z@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed 2021-03-24T20:39:47.507 INFO:teuthology.task.ansible.out:19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-03-24T20:39:47.533 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:47 +0000 (0:00:00.059) 0:00:29.779 ******* 2021-03-24T20:39:48.817 INFO:teuthology.task.ansible.out:FAILED - RETRYING: Clone the keys repo (5 retries left). 2021-03-24T20:39:48.923 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-03-24T20:39:48.925 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 1, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-24T20:39:48.925 INFO:teuthology.task.ansible.out: 2021-03-24T20:39:48.950 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:39:48 +0000 (0:00:01.417) 0:00:31.196 ******* 2021-03-24T20:39:49.221 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:39:49.223 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-24T20:39:49.382 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-24T20:39:49.513 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-24T20:39:49.673 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-03-24T20:39:49.675 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-03-24T20:39:49.822 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-24T20:39:49.972 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-03-24T20:39:49.974 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-03-24T20:39:50.111 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-03-24T20:39:50.249 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-03-24T20:39:50.392 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-03-24T20:39:50.404 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:39:50.406 INFO:teuthology.task.ansible.out: 2021-03-24T20:39:50.567 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-24T20:39:50.569 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-03-24T20:39:50.708 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-24T20:39:50.710 INFO:teuthology.task.ansible.out:son", "validate_certs": true} 2021-03-24T20:39:50.720 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:39:50.722 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:39:50.871 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-03-24T20:39:51.021 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-24T20:39:51.023 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-24T20:39:51.159 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-24T20:39:51.297 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-24T20:39:51.299 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-03-24T20:39:51.446 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-03-24T20:39:51.585 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-24T20:39:51.726 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-24T20:39:51.875 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-03-24T20:39:51.877 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-03-24T20:39:52.014 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-24T20:39:52.016 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-03-24T20:39:52.152 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-24T20:39:52.301 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-24T20:39:52.440 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-24T20:39:52.442 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-24T20:39:52.589 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-24T20:39:52.728 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-24T20:39:52.879 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-24T20:39:53.017 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-24T20:39:53.149 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-24T20:39:53.287 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-24T20:39:53.289 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-03-24T20:39:53.456 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-03-24T20:39:53.594 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-03-24T20:39:53.733 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-03-24T20:39:53.870 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-03-24T20:39:53.997 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-24T20:39:54.146 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-24T20:39:54.304 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-24T20:39:54.443 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-24T20:39:54.583 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-24T20:39:54.720 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-24T20:39:54.722 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-24T20:39:54.858 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-24T20:39:54.861 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-24T20:39:54.986 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-24T20:39:55.127 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-24T20:39:55.269 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-24T20:39:55.420 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-24T20:39:55.422 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-24T20:39:55.560 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-24T20:39:55.562 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-24T20:39:55.564 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-24T20:39:55.699 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-24T20:39:55.826 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-24T20:39:55.975 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-24T20:39:56.112 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-24T20:39:56.239 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-24T20:39:56.384 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-24T20:39:56.386 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-03-24T20:39:56.512 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-24T20:39:56.652 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-24T20:39:56.790 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-24T20:39:56.792 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-03-24T20:39:56.940 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-24T20:39:57.067 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-24T20:39:57.205 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-24T20:39:57.207 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-03-24T20:39:57.343 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-03-24T20:39:57.470 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-24T20:39:57.600 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-03-24T20:39:57.748 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-24T20:39:57.878 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-03-24T20:39:57.880 INFO:teuthology.task.ansible.out: 2021-03-24T20:39:58.020 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-03-24T20:39:58.157 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-03-24T20:39:58.296 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-03-24T20:39:58.463 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-03-24T20:39:58.606 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-03-24T20:39:58.754 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-03-24T20:39:58.935 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-24T20:39:59.074 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-03-24T20:39:59.235 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-03-24T20:39:59.385 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-03-24T20:39:59.523 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-03-24T20:39:59.655 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-24T20:39:59.803 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-24T20:39:59.954 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-24T20:40:00.051 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-03-24T20:40:00.053 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"after": "c31004c4720a55ac12a95d66c32e64f592877604", "attempts": 2, "before": "c31004c4720a55ac12a95d66c32e64f592877604", "changed": false, "remote_url_changed": false} 2021-03-24T20:40:00.078 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:00 +0000 (0:00:11.127) 0:00:42.323 ******* 2021-03-24T20:40:00.098 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:00.099 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-24T20:40:00.240 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-03-24T20:40:00.241 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-24T20:40:00.366 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-03-24T20:40:00.377 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-24T20:40:00.513 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-03-24T20:40:00.515 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-24T20:40:00.642 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-24T20:40:00.654 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-03-24T20:40:00.782 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-03-24T20:40:00.806 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-03-24T20:40:00.808 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-03-24T20:40:00.947 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-03-24T20:40:00.949 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-03-24T20:40:00.951 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-03-24T20:40:01.084 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-24T20:40:01.087 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-03-24T20:40:01.108 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-03-24T20:40:01.110 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-03-24T20:40:01.240 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-24T20:40:01.252 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-03-24T20:40:01.383 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-24T20:40:01.396 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-03-24T20:40:01.525 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-24T20:40:01.558 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-03-24T20:40:01.570 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:01.676 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-03-24T20:40:01.709 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-03-24T20:40:01.711 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-03-24T20:40:01.837 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-03-24T20:40:01.860 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-03-24T20:40:01.861 INFO:teuthology.task.ansible.out:son", "validate_certs": true} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:01.872 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:01.979 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-03-24T20:40:02.012 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-03-24T20:40:02.120 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-24T20:40:02.153 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-03-24T20:40:02.155 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-03-24T20:40:02.271 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-03-24T20:40:02.274 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-03-24T20:40:02.305 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-03-24T20:40:02.414 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-24T20:40:02.459 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": 2021-03-24T20:40:02.461 INFO:teuthology.task.ansible.out:null, "state": "present", "user": "davidz", "validate_certs": true} 2021-03-24T20:40:02.558 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-24T20:40:02.602 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-03-24T20:40:02.710 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-03-24T20:40:02.753 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-03-24T20:40:02.849 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-24T20:40:02.882 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-03-24T20:40:02.992 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-24T20:40:03.028 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-03-24T20:40:03.029 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-03-24T20:40:03.157 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-03-24T20:40:03.169 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-03-24T20:40:03.171 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-03-24T20:40:03.301 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-03-24T20:40:03.303 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-24T20:40:03.313 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-03-24T20:40:03.433 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-24T20:40:03.456 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-03-24T20:40:03.458 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:03.583 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-03-24T20:40:03.606 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-03-24T20:40:03.608 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-03-24T20:40:03.721 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-24T20:40:03.753 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-03-24T20:40:03.861 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-24T20:40:03.893 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-03-24T20:40:04.000 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-03-24T20:40:04.034 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-03-24T20:40:04.140 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-03-24T20:40:04.174 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-03-24T20:40:04.280 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-03-24T20:40:04.313 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-03-24T20:40:04.420 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-03-24T20:40:04.453 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-03-24T20:40:04.456 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-03-24T20:40:04.582 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-03-24T20:40:04.583 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-03-24T20:40:04.733 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-24T20:40:04.806 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-03-24T20:40:04.875 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-24T20:40:04.877 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-03-24T20:40:05.015 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-24T20:40:05.080 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-03-24T20:40:05.155 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-24T20:40:05.168 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-03-24T20:40:05.298 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-24T20:40:05.309 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-03-24T20:40:05.446 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-24T20:40:05.448 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-24T20:40:05.625 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-24T20:40:05.731 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-03-24T20:40:05.753 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-24T20:40:05.902 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-03-24T20:40:05.904 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-24T20:40:06.030 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-03-24T20:40:06.032 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-03-24T20:40:06.042 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-03-24T20:40:06.044 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-03-24T20:40:06.178 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-03-24T20:40:06.180 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-03-24T20:40:06.190 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-24T20:40:06.311 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-03-24T20:40:06.326 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-24T20:40:06.442 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-03-24T20:40:06.455 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-24T20:40:06.580 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-03-24T20:40:06.582 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-24T20:40:06.706 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-24T20:40:06.718 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-03-24T20:40:06.720 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-24T20:40:06.834 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-03-24T20:40:06.845 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-03-24T20:40:06.847 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-03-24T20:40:06.848 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-03-24T20:40:06.963 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-24T20:40:06.974 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-03-24T20:40:07.101 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-24T20:40:07.103 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-03-24T20:40:07.228 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-03-24T20:40:07.239 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-24T20:40:07.366 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-03-24T20:40:07.367 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-24T20:40:07.494 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-24T20:40:07.505 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-03-24T20:40:07.627 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-24T20:40:07.639 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-03-24T20:40:07.641 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-03-24T20:40:07.767 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-03-24T20:40:07.778 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-24T20:40:07.905 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-24T20:40:07.916 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-03-24T20:40:08.043 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-03-24T20:40:08.044 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-03-24T20:40:08.054 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-24T20:40:08.182 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-03-24T20:40:08.193 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-24T20:40:08.310 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-03-24T20:40:08.332 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", " 2021-03-24T20:40:08.334 INFO:teuthology.task.ansible.out:key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-03-24T20:40:08.450 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-03-24T20:40:08.451 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-03-24T20:40:08.461 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-24T20:40:08.547 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:08 +0000 (0:00:08.468) 0:00:50.792 ******* 2021-03-24T20:40:08.577 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-03-24T20:40:08.641 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-24T20:40:08.642 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:08.652 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:08.663 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:08.665 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:08.675 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:08.676 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:08.686 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:08.697 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:08.699 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:08.709 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:08.710 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-03-24T20:40:08.835 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-24T20:40:08.837 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-03-24T20:40:08.846 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:08.848 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nm 2021-03-24T20:40:08.850 INFO:teuthology.task.ansible.out:benjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:08.974 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:08.976 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-03-24T20:40:08.986 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-24T20:40:08.988 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-03-24T20:40:09.114 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:09.118 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** ok: [smithi063.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82d 2021-03-24T20:40:09.120 INFO:teuthology.task.ansible.out:ac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-03-24T20:40:09.126 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.137 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.139 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.149 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.151 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.160 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.171 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.173 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.183 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.185 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.195 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.197 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.207 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.209 INFO:teuthology.task.ansible.out: skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.219 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.220 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:09.222 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.233 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.245 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.249 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.259 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.261 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.272 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.274 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.285 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.287 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.296 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.298 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.309 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.319 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.330 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.332 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.342 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.354 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.365 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.378 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.379 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.389 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.400 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:09.402 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-03-24T20:40:09.402 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.404 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.414 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.416 INFO:teuthology.task.ansible.out: skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.426 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.437 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.448 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.449 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.460 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.474 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.475 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.486 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.498 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.511 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.513 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.523 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.535 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.547 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.549 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-03-24T20:40:09.550 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.562 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.574 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.576 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.586 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.597 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.599 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.609 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.611 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.621 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.622 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.632 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.643 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.646 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.655 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.666 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.668 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.678 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.680 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.690 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.701 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.703 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.713 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.714 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.725 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.727 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:09.737 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.748 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.759 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.762 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.771 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.783 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.785 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:09.794 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.797 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.806 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.808 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:09.817 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.833 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-03-24T20:40:09.846 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-24T20:40:09.848 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.858 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.871 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.881 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.886 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.893 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.904 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.916 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.919 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.928 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.930 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.940 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.942 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.954 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.965 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.967 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.977 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.979 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:09.989 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:10.000 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:10.003 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:10.013 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:10.015 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:10.024 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:10.035 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:10.037 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:10.047 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:10.059 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:10.070 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:10.081 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:10.093 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:10.158 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:10 +0000 (0:00:01.610) 0:00:52.403 ******* 2021-03-24T20:40:10.166 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:10.168 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-03-24T20:40:10.245 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:10.271 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:10 +0000 (0:00:00.113) 0:00:52.516 ******* 2021-03-24T20:40:10.280 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:10.282 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-03-24T20:40:10.400 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-03-24T20:40:10.496 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:10.498 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-03-24T20:40:10.541 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:10.543 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-03-24T20:40:10.638 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:10.640 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-03-24T20:40:10.681 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:10.683 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-03-24T20:40:10.776 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:10.778 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-24T20:40:10.808 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:10.810 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-03-24T20:40:10.923 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi063.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-03-24T20:40:10.945 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:10.947 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-03-24T20:40:11.055 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:11.057 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-24T20:40:11.077 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:11.079 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-03-24T20:40:11.207 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:11.209 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-24T20:40:11.219 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:11.221 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-03-24T20:40:11.346 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:11.348 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-24T20:40:11.357 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:11.359 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-03-24T20:40:11.361 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-03-24T20:40:11.488 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:11.489 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-03-24T20:40:11.501 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:11.502 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-03-24T20:40:11.629 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:11.631 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-03-24T20:40:11.781 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-03-24T20:40:11.783 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi063.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-24T20:40:11.924 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:11.926 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} TASK [Remove revoked users] **************************************************** ok: [smithi063.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-24T20:40:12.077 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc 2021-03-24T20:40:12.079 INFO:teuthology.task.ansible.out:+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.localdomain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} 2021-03-24T20:40:12.089 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:12.091 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-24T20:40:12.217 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:12.219 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQEEJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq 2021-03-24T20:40:12.221 INFO:teuthology.task.ansible.out:2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} 2021-03-24T20:40:12.239 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:12.241 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-24T20:40:12.368 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:12.370 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-03-24T20:40:12.379 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:12.381 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-24T20:40:12.508 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-03-24T20:40:12.530 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:12.532 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-24T20:40:12.657 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:12.659 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-03-24T20:40:12.668 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:12.670 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-24T20:40:12.805 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:12.808 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} TASK [Remove revoked users] **************************************************** ok: [smithi063.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-24T20:40:12.947 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:12.949 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-03-24T20:40:12.969 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:12.971 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-24T20:40:13.108 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:13.110 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-24T20:40:13.252 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-03-24T20:40:13.264 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi063.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-24T20:40:13.395 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:13.396 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-03-24T20:40:13.398 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-03-24T20:40:13.439 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:13.439 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-24T20:40:13.547 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:13.549 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-03-24T20:40:13.580 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:13.582 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-24T20:40:13.676 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:13.678 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-03-24T20:40:13.733 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:13.735 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-24T20:40:13.808 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:13.809 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-03-24T20:40:13.878 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:13.880 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-24T20:40:13.951 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-03-24T20:40:14.019 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:14.021 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-24T20:40:14.083 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:14.085 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-03-24T20:40:14.157 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:14.159 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-24T20:40:14.211 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:14.344 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} TASK [Remove revoked users] **************************************************** ok: [smithi063.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-24T20:40:14.362 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@ 2021-03-24T20:40:14.365 INFO:teuthology.task.ansible.out:redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-03-24T20:40:14.459 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:14.461 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-24T20:40:14.481 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:14.483 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-03-24T20:40:14.598 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:14.602 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-24T20:40:14.630 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:14.632 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-03-24T20:40:14.737 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:14.739 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-24T20:40:14.759 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:14.761 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-03-24T20:40:14.887 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:14.888 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-24T20:40:14.909 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:14.910 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-03-24T20:40:15.057 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-03-24T20:40:15.059 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi063.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-24T20:40:15.175 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:15.177 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-03-24T20:40:15.207 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:15.209 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-24T20:40:15.318 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:15.320 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-03-24T20:40:15.351 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi063.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-24T20:40:15.467 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:15.469 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-03-24T20:40:15.500 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:15.502 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-24T20:40:15.608 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:15.701 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} TASK [Remove revoked users] **************************************************** ok: [smithi063.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-24T20:40:15.748 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:15.749 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-03-24T20:40:15.791 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:15.793 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-24T20:40:15.865 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:15.867 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-03-24T20:40:15.968 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi063.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-03-24T20:40:16.163 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-03-24T20:40:16.174 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:16.176 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-03-24T20:40:16.329 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:16.331 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-24T20:40:16.404 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-03-24T20:40:16.507 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:16.509 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-03-24T20:40:16.596 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:16.598 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-03-24T20:40:16.671 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:16.673 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-24T20:40:16.737 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:16.739 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-03-24T20:40:16.817 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi063.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-24T20:40:16.861 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-03-24T20:40:16.960 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** ok: [smithi063.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-24T20:40:17.024 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:17.026 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-03-24T20:40:17.110 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:17.112 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-24T20:40:17.167 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:17.169 INFO:teuthology.task.ansible.out:TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-03-24T20:40:17.264 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:17.265 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-24T20:40:17.307 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB 2021-03-24T20:40:17.309 INFO:teuthology.task.ansible.out:7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjLd+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-03-24T20:40:17.407 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:17.409 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-24T20:40:17.449 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:17.451 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-03-24T20:40:17.550 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:17.552 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-03-24T20:40:17.593 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:17.595 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-03-24T20:40:17.693 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:17.695 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-24T20:40:17.770 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:17 +0000 (0:00:07.499) 0:01:00.016 ******* 2021-03-24T20:40:17.778 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:17.780 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-03-24T20:40:17.914 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-03-24T20:40:18.065 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-03-24T20:40:18.194 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-03-24T20:40:18.330 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-03-24T20:40:18.337 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616618417.8964746-26093-172679425948399/source", "state": "file", "uid": 0} 2021-03-24T20:40:18.338 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:18.338 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-03-24T20:40:18.367 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:18 +0000 (0:00:00.596) 0:01:00.613 ******* 2021-03-24T20:40:18.458 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-03-24T20:40:18.460 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => { "msg": "Host smithi063.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-03-24T20:40:18.486 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:18 +0000 (0:00:00.119) 0:01:00.732 ******* 2021-03-24T20:40:18.493 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:18.495 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-03-24T20:40:18.604 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-03-24T20:40:18.605 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-03-24T20:40:18.631 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:18 +0000 (0:00:00.144) 0:01:00.876 ******* 2021-03-24T20:40:18.639 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:18.641 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-03-24T20:40:18.716 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:18.739 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:18 +0000 (0:00:00.108) 0:01:00.985 ******* 2021-03-24T20:40:18.767 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:18.768 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-03-24T20:40:18.820 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:18.856 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:18 +0000 (0:00:00.106) 0:01:01.092 ******* 2021-03-24T20:40:18.907 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:18.909 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-03-24T20:40:18.962 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-03-24T20:40:18.964 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:18.992 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:18 +0000 (0:00:00.145) 0:01:01.238 ******* 2021-03-24T20:40:19.059 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:19.060 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-03-24T20:40:19.091 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:19.117 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:19 +0000 (0:00:00.124) 0:01:01.362 ******* 2021-03-24T20:40:19.215 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-03-24T20:40:19.217 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys using the keys repo] ********************** ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-03-24T20:40:19.245 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:19 +0000 (0:00:00.128) 0:01:01.491 ******* 2021-03-24T20:40:19.315 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:19.341 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:19 +0000 (0:00:00.096) 0:01:01.587 ******* 2021-03-24T20:40:19.359 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:19.360 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-03-24T20:40:19.426 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** 2021-03-24T20:40:19.428 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:19.450 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:19 +0000 (0:00:00.108) 0:01:01.696 ******* 2021-03-24T20:40:19.498 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:19.500 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-03-24T20:40:19.521 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:19.523 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:19.544 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:19 +0000 (0:00:00.094) 0:01:01.790 ******* 2021-03-24T20:40:19.635 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-03-24T20:40:19.637 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi063.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T20:40:19.667 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:19 +0000 (0:00:00.122) 0:01:01.913 ******* 2021-03-24T20:40:19.675 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:19.677 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-03-24T20:40:19.755 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:19.783 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:19 +0000 (0:00:00.116) 0:01:02.029 ******* 2021-03-24T20:40:19.802 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:19.804 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ3ch+pU88/tX5TVVFl/1pPief8drYLRkK4BtfJk+xVNxX9qkDtOtiLAU+OugYF1/xk9hE9zTbm3UBL23unrzhlR80t8f2FiF1YFhHVzeZ/p/OunmkU0RFQzkMVuwKVOUNulw9FwpEZ/0Bjfxvaims0SBWaCzBgE5g0R+Ofdq/bVF82K8vU5Laa3KZ/cCdGzFgW00cBvPAKIpQ93g8TRqsCsFvR37nY9v5SU1hJ9araEp6LRA86tUt3EFtxD2tRcmqKGtCb0kOr+rRXWVGiaUBiedHxFudIy0+TNj4fNrJwpoC0hcBO9ek4Bh6hM7Zqzum9HpkPGlz84jkm+r+xMThaZCh8iFkgxyK+bw8B8smaVJD04ob9Wb4RYoFYfQ7TEkPkO9gr+UsSesbP9fWfucemUqwK0htOOtpc1YRwdOPcdPJWAfx9J5+YVL8BTWYBaTdmHslOKeUGM5fCob6QxC6Er9xweys9jh7MXttSpe013bY7P4ZDRF8uWvygpsytPE= amathuri@redhat.com", " 2021-03-24T20:40:19.805 INFO:teuthology.task.ansible.out:key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-03-24T20:40:19.857 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:19.881 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:19 +0000 (0:00:00.097) 0:01:02.126 ******* 2021-03-24T20:40:19.929 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-03-24T20:40:19.931 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-03-24T20:40:20.002 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-03-24T20:40:20.004 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T20:40:20.025 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:20 +0000 (0:00:00.144) 0:01:02.271 ******* 2021-03-24T20:40:20.048 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:20 +0000 (0:00:00.022) 0:01:02.293 ******* 2021-03-24T20:40:20.132 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-03-24T20:40:20.133 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.134 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.164 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:20 +0000 (0:00:00.116) 0:01:02.409 ******* 2021-03-24T20:40:20.172 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.173 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.175 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.176 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.241 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-03-24T20:40:20.243 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T20:40:20.264 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:20 +0000 (0:00:00.100) 0:01:02.510 ******* 2021-03-24T20:40:20.305 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-24T20:40:20.307 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-03-24T20:40:20.310 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.321 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.346 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.372 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:20 +0000 (0:00:00.107) 0:01:02.618 ******* 2021-03-24T20:40:20.481 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-24T20:40:20.483 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} TASK [common : List CDN releases available to system] ************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was Fal 2021-03-24T20:40:20.486 INFO:teuthology.task.ansible.out:se"} 2021-03-24T20:40:20.506 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:20 +0000 (0:00:00.134) 0:01:02.752 ******* 2021-03-24T20:40:20.597 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-03-24T20:40:20.599 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.629 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:20 +0000 (0:00:00.122) 0:01:02.874 ******* 2021-03-24T20:40:20.640 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-24T20:40:20.641 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-03-24T20:40:20.643 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfad 2021-03-24T20:40:20.644 INFO:teuthology.task.ansible.out:a28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.665 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.677 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.679 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\ndavid@ubuntu-2004 sz2UXQopIP7TJ3t/XtOylw 515687a2ace008dbc318525cb273afed0887aa03ae9ca9b0933ca4792ef4addc\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.689 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.701 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.704 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.713 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.724 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.735 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.737 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.747 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-24T20:40:20.749 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.783 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:20 +0000 (0:00:00.154) 0:01:03.029 ******* 2021-03-24T20:40:20.794 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.795 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.796 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.797 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.797 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJi 2021-03-24T20:40:20.799 INFO:teuthology.task.ansible.out:hFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.810 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.821 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.832 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.843 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.855 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.858 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.868 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.879 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-03-24T20:40:20.880 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.912 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:20 +0000 (0:00:00.128) 0:01:03.157 ******* 2021-03-24T20:40:20.924 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.925 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.926 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.927 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.927 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.939 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.941 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.951 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.962 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.974 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.974 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.986 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.997 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:20.999 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.025 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:21 +0000 (0:00:00.113) 0:01:03.271 ******* 2021-03-24T20:40:21.033 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.060 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.061 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.062 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.062 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 2021-03-24T20:40:21.064 INFO:teuthology.task.ansible.out: f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.074 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.085 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.096 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.101 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.108 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.119 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.121 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.144 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:21 +0000 (0:00:00.118) 0:01:03.389 ******* 2021-03-24T20:40:21.151 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-24T20:40:21.152 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.152 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.164 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.175 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.187 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.188 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.198 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.212 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} TASK [common : Store list of enabled repos] ************************************ skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.235 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:21 +0000 (0:00:00.091) 0:01:03.481 ******* 2021-03-24T20:40:21.242 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-24T20:40:21.244 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.255 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.257 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.267 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.269 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.279 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.281 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.291 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.302 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.304 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.331 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:21 +0000 (0:00:00.095) 0:01:03.576 ******* 2021-03-24T20:40:21.339 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.341 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.342 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.345 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.346 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.358 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.369 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.371 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.381 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.393 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.394 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.396 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.425 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:21 +0000 (0:00:00.093) 0:01:03.670 ******* 2021-03-24T20:40:21.435 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-03-24T20:40:21.437 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.437 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.439 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.439 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.440 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.451 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.463 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.466 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.475 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.476 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.488 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.490 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.500 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-03-24T20:40:21.503 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.532 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:21 +0000 (0:00:00.107) 0:01:03.778 ******* 2021-03-24T20:40:21.542 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.543 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.544 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.545 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.545 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.545 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.557 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.569 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.571 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.582 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.584 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.594 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.595 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.605 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.617 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.618 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.628 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-03-24T20:40:21.630 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} TASK [users : Update authorized_keys for each user with literal keys] ********** skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.653 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:21 +0000 (0:00:00.120) 0:01:03.899 ******* 2021-03-24T20:40:21.664 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.665 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267\namathuri@lenovothinkpad MhlrEG4X+iKq2d2GxwMbmA 812cfea500340140144ecb27f7851b86d8daff658f173189fceec8e61b59cbf2\n"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.667 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.716 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:21 +0000 (0:00:00.062) 0:01:03.962 ******* 2021-03-24T20:40:21.734 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-03-24T20:40:21.736 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.761 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:21 +0000 (0:00:00.044) 0:01:04.006 ******* 2021-03-24T20:40:21.820 INFO:teuthology.task.ansible.out: TASK [Filter the revoked_users list] ******************************************* 2021-03-24T20:40:21.821 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.847 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:21 +0000 (0:00:00.086) 0:01:04.093 ******* 2021-03-24T20:40:21.871 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:21.909 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:21 +0000 (0:00:00.061) 0:01:04.154 ******* 2021-03-24T20:40:22.006 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:22.030 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:22 +0000 (0:00:00.120) 0:01:04.275 ******* 2021-03-24T20:40:22.162 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} TASK [Remove revoked users] **************************************************** ok: [smithi052.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-03-24T20:40:22.171 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-03-24T20:40:22.173 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:22.200 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:22 +0000 (0:00:00.170) 0:01:04.446 ******* 2021-03-24T20:40:22.304 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:22.306 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-03-24T20:40:22.327 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-03-24T20:40:22.328 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} skipping: [smithi063.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:22.352 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:22 +0000 (0:00:00.151) 0:01:04.598 ******* 2021-03-24T20:40:22.354 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:22.446 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:22.479 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:22 +0000 (0:00:00.127) 0:01:04.725 ******* 2021-03-24T20:40:22.490 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:22.491 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-03-24T20:40:22.609 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-03-24T20:40:22.683 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-24T20:40:22.685 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.009267", "end": "2021-03-24 20:40:22.245142", "rc": 0, "start": "2021-03-24 20:40:22.235875", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-24T20:40:22.712 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:22 +0000 (0:00:00.232) 0:01:04.957 ******* 2021-03-24T20:40:22.739 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:22.741 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-03-24T20:40:22.793 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-03-24T20:40:22.794 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:22.816 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:22 +0000 (0:00:00.104) 0:01:05.062 ******* 2021-03-24T20:40:22.878 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:22.880 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-03-24T20:40:22.901 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:22.934 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:22 +0000 (0:00:00.118) 0:01:05.180 ******* 2021-03-24T20:40:23.031 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:23.032 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-03-24T20:40:23.054 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** skipping: [smithi063.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} skipping: [smithi063.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:23.065 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:23.066 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:23.103 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:23 +0000 (0:00:00.168) 0:01:05.349 ******* 2021-03-24T20:40:23.173 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:23.175 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} TASK [common : Get the current timezone] *************************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:23.198 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:23 +0000 (0:00:00.094) 0:01:05.443 ******* 2021-03-24T20:40:23.281 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:23.307 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:23 +0000 (0:00:00.109) 0:01:05.553 ******* 2021-03-24T20:40:23.331 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:23.333 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-03-24T20:40:23.375 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-03-24T20:40:23.377 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:23.398 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:23 +0000 (0:00:00.091) 0:01:05.644 ******* 2021-03-24T20:40:23.460 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:23.462 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-03-24T20:40:23.475 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:23.498 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:23 +0000 (0:00:00.099) 0:01:05.744 ******* 2021-03-24T20:40:23.571 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-03-24T20:40:23.572 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:23.596 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:23 +0000 (0:00:00.097) 0:01:05.841 ******* 2021-03-24T20:40:23.602 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:23.604 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-03-24T20:40:23.731 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-03-24T20:40:23.881 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-03-24T20:40:23.949 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-03-24T20:40:23.951 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.212475", "end": "2021-03-24 20:40:23.500396", "rc": 0, "start": "2021-03-24 20:40:23.287921", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:40:23.977 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:23 +0000 (0:00:00.381) 0:01:06.222 ******* 2021-03-24T20:40:24.037 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:24.040 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-03-24T20:40:24.070 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:24.093 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:24 +0000 (0:00:00.116) 0:01:06.339 ******* 2021-03-24T20:40:24.173 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:24.175 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-03-24T20:40:24.311 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-03-24T20:40:24.345 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ [WARNING]: Consider using the apt module rather than running 'apt-get'. If you need to use command because apt is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.049958", "end": "2021-03-24 20:40:23.894001", "rc": 0, "start": "2021-03-24 20:40:23.844043", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:40:24.372 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:24 +0000 (0:00:00.278) 0:01:06.617 ******* 2021-03-24T20:40:24.452 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:24.453 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-03-24T20:40:24.590 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-03-24T20:40:24.739 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-03-24T20:40:24.887 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-03-24T20:40:25.025 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-03-24T20:40:25.163 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-03-24T20:40:25.301 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-03-24T20:40:25.451 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-03-24T20:40:25.588 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-03-24T20:40:25.738 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-03-24T20:40:25.890 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-03-24T20:40:26.027 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-03-24T20:40:26.165 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-03-24T20:40:26.304 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-03-24T20:40:26.447 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-03-24T20:40:26.585 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-03-24T20:40:26.743 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-03-24T20:40:26.883 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-03-24T20:40:27.021 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-03-24T20:40:27.171 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-03-24T20:40:27.320 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-03-24T20:40:27.458 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": false, "item": "dorinda", "name": "dorinda", "state": "absent"} 2021-03-24T20:40:27.613 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": false, "item": "zyan", "name": "zyan", "state": "absent"} 2021-03-24T20:40:27.758 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-03-24T20:40:27.897 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-03-24T20:40:28.099 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-03-24T20:40:28.248 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-03-24T20:40:28.249 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:28.294 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-03-24T20:40:28.296 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1616618190, "cache_updated": false, "changed": false} 2021-03-24T20:40:28.328 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:28 +0000 (0:00:03.956) 0:01:10.574 ******* 2021-03-24T20:40:28.387 INFO:teuthology.task.ansible.out: TASK [Remove revoked users] **************************************************** 2021-03-24T20:40:28.389 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-03-24T20:40:28.537 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-03-24T20:40:28.677 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-03-24T20:40:28.826 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-03-24T20:40:28.975 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-03-24T20:40:29.113 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-03-24T20:40:29.174 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:29 +0000 (0:00:00.845) 0:01:11.419 ******* 2021-03-24T20:40:29.291 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-03-24T20:40:29.293 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"cache_update_time": 1616618190, "cache_updated": false, "changed": false} 2021-03-24T20:40:29.316 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:29 +0000 (0:00:00.142) 0:01:11.562 ******* 2021-03-24T20:40:29.399 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:29.425 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:29 +0000 (0:00:00.108) 0:01:11.670 ******* 2021-03-24T20:40:29.650 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-03-24T20:40:29.651 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "checksum": "1d9711ff879c231c67cb70d00ca0088797a377d0", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "80a55def4b505d08aff3687f81f74dc2", "mode": "0644", "owner": "root", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616618429.2799513-28590-106413479115276/source", "state": "file", "uid": 0} 2021-03-24T20:40:29.675 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:29 +0000 (0:00:00.249) 0:01:11.920 ******* 2021-03-24T20:40:29.756 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-03-24T20:40:29.758 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => { "msg": "Host smithi052.front.sepia.ceph.com is running Ubuntu 18.04 (bionic)" } 2021-03-24T20:40:29.779 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:29 +0000 (0:00:00.104) 0:01:12.025 ******* 2021-03-24T20:40:29.862 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/empty.yml"} 2021-03-24T20:40:29.884 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:29 +0000 (0:00:00.104) 0:01:12.129 ******* 2021-03-24T20:40:29.965 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:29.989 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:29 +0000 (0:00:00.105) 0:01:12.235 ******* 2021-03-24T20:40:30.074 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-03-24T20:40:30.076 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:30.097 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:30 +0000 (0:00:00.107) 0:01:12.343 ******* 2021-03-24T20:40:30.168 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-03-24T20:40:30.170 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-03-24T20:40:30.201 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:30 +0000 (0:00:00.104) 0:01:12.447 ******* 2021-03-24T20:40:30.208 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-03-24T20:40:30.210 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:30.234 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:30 +0000 (0:00:00.032) 0:01:12.480 ******* 2021-03-24T20:40:30.325 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:30.348 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:30 +0000 (0:00:00.113) 0:01:12.594 ******* 2021-03-24T20:40:30.439 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-03-24T20:40:30.440 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:30.467 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:30 +0000 (0:00:00.118) 0:01:12.712 ******* 2021-03-24T20:40:30.538 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-03-24T20:40:30.539 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:30.562 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:30 +0000 (0:00:00.095) 0:01:12.807 ******* 2021-03-24T20:40:30.634 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:30.659 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:30 +0000 (0:00:00.097) 0:01:12.905 ******* 2021-03-24T20:40:30.730 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-03-24T20:40:30.732 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:30.757 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:30 +0000 (0:00:00.097) 0:01:13.002 ******* 2021-03-24T20:40:30.850 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-03-24T20:40:30.852 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T20:40:30.874 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:30 +0000 (0:00:00.117) 0:01:13.120 ******* 2021-03-24T20:40:30.947 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-03-24T20:40:30.949 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:30.971 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:30 +0000 (0:00:00.096) 0:01:13.217 ******* 2021-03-24T20:40:31.041 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-03-24T20:40:31.043 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:31.065 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:31 +0000 (0:00:00.093) 0:01:13.310 ******* 2021-03-24T20:40:31.066 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:31.137 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-03-24T20:40:31.138 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T20:40:31.160 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:31 +0000 (0:00:00.095) 0:01:13.406 ******* 2021-03-24T20:40:31.168 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-03-24T20:40:31.169 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"cache_update_time": 1616618190, "cache_updated": false, "changed": false} 2021-03-24T20:40:31.196 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:31 +0000 (0:00:00.035) 0:01:13.441 ******* 2021-03-24T20:40:31.235 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-03-24T20:40:31.236 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:31.261 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:31 +0000 (0:00:00.065) 0:01:13.507 ******* 2021-03-24T20:40:31.353 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-03-24T20:40:31.356 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T20:40:31.379 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:31 +0000 (0:00:00.117) 0:01:13.625 ******* 2021-03-24T20:40:31.451 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-03-24T20:40:31.452 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:31.475 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:31 +0000 (0:00:00.095) 0:01:13.720 ******* 2021-03-24T20:40:31.580 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:31.607 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:31 +0000 (0:00:00.132) 0:01:13.853 ******* 2021-03-24T20:40:31.629 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-24T20:40:31.630 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-03-24T20:40:31.674 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:31.699 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:31 +0000 (0:00:00.092) 0:01:13.945 ******* 2021-03-24T20:40:31.771 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-03-24T20:40:31.837 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 24 March 2021 20:40:31 +0000 (0:00:00.095) 0:01:14.040 ******* 2021-03-24T20:40:31.854 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* 2021-03-24T20:40:31.856 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T20:40:31.878 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:31 +0000 (0:00:00.083) 0:01:14.124 ******* 2021-03-24T20:40:31.927 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-24T20:40:31.929 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-03-24T20:40:31.950 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:31.973 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:31 +0000 (0:00:00.094) 0:01:14.219 ******* 2021-03-24T20:40:32.054 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* 2021-03-24T20:40:32.056 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:32.077 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:32 +0000 (0:00:00.103) 0:01:14.323 ******* 2021-03-24T20:40:32.159 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-03-24T20:40:32.161 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:32.183 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:32 +0000 (0:00:00.106) 0:01:14.429 ******* 2021-03-24T20:40:32.266 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-03-24T20:40:32.268 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:32.294 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:32 +0000 (0:00:00.110) 0:01:14.540 ******* 2021-03-24T20:40:32.304 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-03-24T20:40:32.306 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-03-24T20:40:32.363 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:32 +0000 (0:00:00.068) 0:01:14.609 ******* 2021-03-24T20:40:32.383 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:32.408 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:32 +0000 (0:00:00.044) 0:01:14.653 ******* 2021-03-24T20:40:32.477 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-03-24T20:40:32.479 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:32.504 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:32 +0000 (0:00:00.096) 0:01:14.750 ******* 2021-03-24T20:40:32.524 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-03-24T20:40:32.526 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T20:40:32.553 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:32 +0000 (0:00:00.048) 0:01:14.799 ******* 2021-03-24T20:40:32.580 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-03-24T20:40:32.582 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T20:40:32.609 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:32 +0000 (0:00:00.055) 0:01:14.854 ******* 2021-03-24T20:40:32.693 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-03-24T20:40:32.695 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:32.717 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:32 +0000 (0:00:00.108) 0:01:14.963 ******* 2021-03-24T20:40:32.793 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-03-24T20:40:32.795 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:32.818 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:32 +0000 (0:00:00.100) 0:01:15.064 ******* 2021-03-24T20:40:32.887 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-03-24T20:40:32.889 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:32.912 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:32 +0000 (0:00:00.093) 0:01:15.157 ******* 2021-03-24T20:40:32.972 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-24T20:40:32.974 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-03-24T20:40:32.994 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-03-24T20:40:32.996 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:33.006 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:33.032 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:33 +0000 (0:00:00.120) 0:01:15.278 ******* 2021-03-24T20:40:33.114 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:33.116 INFO:teuthology.task.ansible.out:TASK [common : Configure local epel mirrorlists] ******************************* skipping: [smithi052.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:33.127 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "item": "epel-testing-mirrorlist", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:33.152 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:33 +0000 (0:00:00.118) 0:01:15.396 ******* 2021-03-24T20:40:33.237 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-03-24T20:40:33.239 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:33.260 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:33 +0000 (0:00:00.109) 0:01:15.506 ******* 2021-03-24T20:40:33.268 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-24T20:40:33.269 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-03-24T20:40:33.473 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-24T20:40:33.475 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": "cat /etc/timezone", "delta": "0:00:00.011422", "end": "2021-03-24 20:40:33.268314", "rc": 0, "start": "2021-03-24 20:40:33.256892", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-03-24T20:40:33.499 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:33 +0000 (0:00:00.238) 0:01:15.745 ******* 2021-03-24T20:40:33.562 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-24T20:40:33.565 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-03-24T20:40:33.589 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:33 +0000 (0:00:00.090) 0:01:15.835 ******* 2021-03-24T20:40:33.606 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-03-24T20:40:33.608 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:33.635 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:33 +0000 (0:00:00.046) 0:01:15.881 ******* 2021-03-24T20:40:33.676 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-24T20:40:33.678 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-03-24T20:40:33.702 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:33 +0000 (0:00:00.065) 0:01:15.947 ******* 2021-03-24T20:40:33.718 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-03-24T20:40:33.720 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:33.746 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:33 +0000 (0:00:00.044) 0:01:15.992 ******* 2021-03-24T20:40:33.774 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-03-24T20:40:33.775 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:33.800 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:33 +0000 (0:00:00.054) 0:01:16.046 ******* 2021-03-24T20:40:33.818 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** skipping: [smithi052.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:33.829 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:33.831 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:33.841 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:33.881 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:33 +0000 (0:00:00.080) 0:01:16.127 ******* 2021-03-24T20:40:33.891 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-24T20:40:33.893 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T20:40:33.924 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:33 +0000 (0:00:00.042) 0:01:16.169 ******* 2021-03-24T20:40:33.953 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-03-24T20:40:33.954 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:33.978 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:33 +0000 (0:00:00.054) 0:01:16.224 ******* 2021-03-24T20:40:34.009 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-24T20:40:34.011 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:34.025 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:34.037 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:34.039 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:34.050 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:34.075 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:34 +0000 (0:00:00.096) 0:01:16.320 ******* 2021-03-24T20:40:34.081 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** 2021-03-24T20:40:34.083 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:34.110 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:34 +0000 (0:00:00.035) 0:01:16.356 ******* 2021-03-24T20:40:34.112 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:34.151 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:34.187 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:34 +0000 (0:00:00.076) 0:01:16.432 ******* 2021-03-24T20:40:34.193 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-03-24T20:40:34.195 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:34.222 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:34 +0000 (0:00:00.035) 0:01:16.468 ******* 2021-03-24T20:40:34.307 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-03-24T20:40:34.309 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:34.334 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:34 +0000 (0:00:00.111) 0:01:16.579 ******* 2021-03-24T20:40:34.428 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** 2021-03-24T20:40:34.429 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:34.457 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:34 +0000 (0:00:00.122) 0:01:16.702 ******* 2021-03-24T20:40:34.706 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-03-24T20:40:34.708 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.113093", "end": "2021-03-24 20:40:34.500412", "rc": 0, "start": "2021-03-24 20:40:34.387319", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:40:34.734 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:34 +0000 (0:00:00.277) 0:01:16.979 ******* 2021-03-24T20:40:34.809 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-03-24T20:40:34.811 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:34.833 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:34 +0000 (0:00:00.099) 0:01:17.079 ******* 2021-03-24T20:40:35.102 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-03-24T20:40:35.104 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["apt-get", "clean"], "delta": "0:00:00.040251", "end": "2021-03-24 20:40:34.886425", "rc": 0, "start": "2021-03-24 20:40:34.846174", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:40:35.127 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:35 +0000 (0:00:00.293) 0:01:17.372 ******* 2021-03-24T20:40:35.176 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-03-24T20:40:35.177 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"cache_update_time": 1616618190, "cache_updated": false, "changed": false} 2021-03-24T20:40:35.202 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:35 +0000 (0:00:00.075) 0:01:17.448 ******* 2021-03-24T20:40:35.595 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-03-24T20:40:35.597 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-03-24T20:40:35.621 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:35 +0000 (0:00:00.419) 0:01:17.867 ******* 2021-03-24T20:40:36.027 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:36.028 INFO:teuthology.task.ansible.out:TASK [common : Upload nagios check_mem script] ********************************* ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-03-24T20:40:36.054 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:36 +0000 (0:00:00.432) 0:01:18.299 ******* 2021-03-24T20:40:36.255 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-24T20:40:36.257 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} 2021-03-24T20:40:36.377 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-03-24T20:40:36.401 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:36 +0000 (0:00:00.347) 0:01:18.647 ******* 2021-03-24T20:40:36.503 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-03-24T20:40:36.505 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:36.536 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:36 +0000 (0:00:00.134) 0:01:18.782 ******* 2021-03-24T20:40:36.629 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-03-24T20:40:36.631 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T20:40:36.656 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:36 +0000 (0:00:00.119) 0:01:18.902 ******* 2021-03-24T20:40:36.727 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-03-24T20:40:36.728 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:36.757 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:36 +0000 (0:00:00.101) 0:01:19.003 ******* 2021-03-24T20:40:37.292 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-03-24T20:40:37.294 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-03-24T20:40:37.326 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:37 +0000 (0:00:00.568) 0:01:19.571 ******* 2021-03-24T20:40:38.033 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-03-24T20:40:38.035 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-03-24 20:39:07 UTC", "ActiveEnterTimestampMonotonic": "18580966", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "remote-fs.target var-run.mount basic.target network.target time-sync.target -.mount local-fs.target systemd-tmpfiles-setup.service sysinit.target nss-lookup.target systemd-journald.socket system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-03-24 20:39:07 UTC", "AssertTimestampMonotonic": "18579836", "Before": "shutdown.target multi-user.target xdm.service plymouth-quit.service getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-03-24 20:39:07 UTC", "ConditionTimestampMonotonic": "18579833", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-03-24T20:40:38.037 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1389", "ExecMainStartTimestamp": "Wed 2021-03-24 20:39:07 UTC", "ExecMainStartTimestampMonotonic": "18580922", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Wed 2021-03-24 20:39:07 UTC] ; stop_time=[n/a] ; pid=1389 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-03-24 20:39:07 UTC", "InactiveExitTimestampMonotonic": "18580966", "InvocationID": "629e8c7c4e8e4f8688b39c25b046d1a6", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2021-03-24T20:40:38.039 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1389", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice sysinit.target -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2021-03-24T20:40:38.040 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-03-24 20:39:07 UTC", "StateChangeTimestampMonotonic": "18580966", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-03-24 20:39:07 UTC", "WatchdogTimestampMonotonic": "18580962", "WatchdogUSec": "0"}} 2021-03-24T20:40:38.069 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:38 +0000 (0:00:00.743) 0:01:20.315 ******* 2021-03-24T20:40:38.326 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-03-24T20:40:38.328 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:38.354 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:38 +0000 (0:00:00.285) 0:01:20.600 ******* 2021-03-24T20:40:38.424 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-03-24T20:40:38.426 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:38.449 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:38 +0000 (0:00:00.093) 0:01:20.694 ******* 2021-03-24T20:40:38.552 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:38.589 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:38 +0000 (0:00:00.140) 0:01:20.835 ******* 2021-03-24T20:40:38.682 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-03-24T20:40:38.684 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:38.708 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:38 +0000 (0:00:00.119) 0:01:20.954 ******* 2021-03-24T20:40:38.789 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-03-24T20:40:38.791 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:38.816 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:38 +0000 (0:00:00.107) 0:01:21.062 ******* 2021-03-24T20:40:38.887 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-03-24T20:40:38.889 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:38.913 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:38 +0000 (0:00:00.096) 0:01:21.158 ******* 2021-03-24T20:40:38.995 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-03-24T20:40:38.996 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:39.018 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:39 +0000 (0:00:00.105) 0:01:21.264 ******* 2021-03-24T20:40:39.089 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** ok: [smithi052.front.sepia.ceph.com] => {"attempts": 1, "cache_update_time": 1616618145, "cache_updated": false, "changed": false} TASK [common : nrpe - Build SELinux policy package] **************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:39.113 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:39 +0000 (0:00:00.095) 0:01:21.359 ******* 2021-03-24T20:40:39.137 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:39 +0000 (0:00:00.023) 0:01:21.383 ******* 2021-03-24T20:40:39.210 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-03-24T20:40:39.212 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:39.234 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:39 +0000 (0:00:00.096) 0:01:21.479 ******* 2021-03-24T20:40:39.317 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-03-24T20:40:39.318 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:39.344 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:39 +0000 (0:00:00.109) 0:01:21.589 ******* 2021-03-24T20:40:39.575 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-03-24T20:40:39.577 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-03-24T20:40:39.601 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:39 +0000 (0:00:00.257) 0:01:21.846 ******* 2021-03-24T20:40:39.840 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-03-24T20:40:39.842 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-03-24T20:40:39.866 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:39 +0000 (0:00:00.265) 0:01:22.112 ******* 2021-03-24T20:40:40.081 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-03-24T20:40:40.083 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1616618145, "cache_updated": false, "changed": false} 2021-03-24T20:40:40.106 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:40 +0000 (0:00:00.240) 0:01:22.352 ******* 2021-03-24T20:40:40.158 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-03-24T20:40:40.159 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-03-24T20:40:40.193 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:40 +0000 (0:00:00.086) 0:01:22.439 ******* 2021-03-24T20:40:40.200 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-03-24T20:40:40.202 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:40.343 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:40 +0000 (0:00:00.150) 0:01:22.589 ******* 2021-03-24T20:40:40.404 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-03-24T20:40:40.406 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-24T20:40:40.432 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:40 +0000 (0:00:00.088) 0:01:22.678 ******* 2021-03-24T20:40:40.599 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-03-24T20:40:40.601 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-24T20:40:40.624 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:40 +0000 (0:00:00.191) 0:01:22.869 ******* 2021-03-24T20:40:40.788 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-03-24T20:40:40.790 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "size": 131, "state": "file", "uid": 0} 2021-03-24T20:40:40.815 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:40 +0000 (0:00:00.191) 0:01:23.061 ******* 2021-03-24T20:40:40.843 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-03-24T20:40:40.845 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-24T20:40:40.869 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:40 +0000 (0:00:00.053) 0:01:23.115 ******* 2021-03-24T20:40:40.948 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-03-24T20:40:40.949 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:40.972 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:40 +0000 (0:00:00.103) 0:01:23.218 ******* 2021-03-24T20:40:41.063 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-03-24T20:40:41.065 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:41.090 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:41 +0000 (0:00:00.117) 0:01:23.336 ******* 2021-03-24T20:40:41.311 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** ok: [smithi063.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-03-24T20:40:41.336 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:41 +0000 (0:00:00.246) 0:01:23.582 ******* 2021-03-24T20:40:41.809 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-24T20:40:41.811 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} TASK [common : Make sure smartmontools is installed] *************************** ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1616618145, "cache_updated": false, "changed": false} 2021-03-24T20:40:41.834 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:41 +0000 (0:00:00.497) 0:01:24.080 ******* 2021-03-24T20:40:41.863 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:41 +0000 (0:00:00.028) 0:01:24.109 ******* 2021-03-24T20:40:41.948 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-03-24T20:40:41.950 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi063"}, "changed": false} 2021-03-24T20:40:41.972 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:41 +0000 (0:00:00.108) 0:01:24.218 ******* 2021-03-24T20:40:42.253 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** ok: [smithi052.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "size": 2716736, "state": "file", "uid": 0} 2021-03-24T20:40:42.575 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "size": 1952038, "state": "file", "uid": 0} 2021-03-24T20:40:42.877 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "size": 580432, "state": "file", "uid": 0} 2021-03-24T20:40:42.908 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:42 +0000 (0:00:00.936) 0:01:25.154 ******* 2021-03-24T20:40:42.916 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-03-24T20:40:42.918 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi063.front.sepia.ceph.com", "ansible_hostname": "smithi063", "ansible_nodename": "smithi063"}, "changed": true, "name": "smithi063"} 2021-03-24T20:40:42.945 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:42 +0000 (0:00:00.036) 0:01:25.190 ******* 2021-03-24T20:40:43.047 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-03-24T20:40:43.050 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T20:40:43.077 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:43 +0000 (0:00:00.128) 0:01:25.319 ******* 2021-03-24T20:40:43.361 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-03-24T20:40:43.363 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-03-24T20:40:43.394 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:43 +0000 (0:00:00.320) 0:01:25.640 ******* 2021-03-24T20:40:43.519 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-24T20:40:43.560 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": false, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "mode": "0755", "owner": "root", "path": "/usr/libexec/smart.sh", "size": 13355, "state": "file", "uid": 0} 2021-03-24T20:40:43.809 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-03-24T20:40:43.839 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:43 +0000 (0:00:00.444) 0:01:26.085 ******* 2021-03-24T20:40:43.846 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-03-24T20:40:43.847 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "size": 8602, "state": "file", "uid": 0} 2021-03-24T20:40:44.143 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "size": 3605, "state": "file", "uid": 0} 2021-03-24T20:40:44.185 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:44 +0000 (0:00:00.330) 0:01:26.415 ******* 2021-03-24T20:40:44.251 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-03-24T20:40:44.253 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib/nagios/plugins", "nrpe_group": "nagios", "nrpe_packages": ["nagios-nrpe-server", "nagios-plugins-basic"], "nrpe_service_name": "nagios-nrpe-server", "nrpe_user": "nagios"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/apt_systems.yml"], "changed": false} 2021-03-24T20:40:44.276 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:44 +0000 (0:00:00.106) 0:01:26.522 ******* 2021-03-24T20:40:44.356 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-03-24T20:40:44.358 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:44.380 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:44 +0000 (0:00:00.103) 0:01:26.625 ******* 2021-03-24T20:40:44.387 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-03-24T20:40:44.388 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-24T20:40:44.414 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:44 +0000 (0:00:00.033) 0:01:26.659 ******* 2021-03-24T20:40:44.453 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-03-24T20:40:44.455 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T20:40:44.480 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:44 +0000 (0:00:00.065) 0:01:26.725 ******* 2021-03-24T20:40:44.496 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-03-24T20:40:44.498 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:44.525 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:44 +0000 (0:00:00.045) 0:01:26.771 ******* 2021-03-24T20:40:44.573 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-03-24T20:40:44.575 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:44.585 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:44.587 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:44.613 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:44 +0000 (0:00:00.088) 0:01:26.859 ******* 2021-03-24T20:40:44.620 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-24T20:40:44.622 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:44.646 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:44 +0000 (0:00:00.033) 0:01:26.892 ******* 2021-03-24T20:40:44.684 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-03-24T20:40:44.686 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:44.709 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:44 +0000 (0:00:00.062) 0:01:26.954 ******* 2021-03-24T20:40:44.729 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:44.769 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:44 +0000 (0:00:00.047) 0:01:27.002 ******* 2021-03-24T20:40:44.837 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-24T20:40:44.839 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:44.861 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:44 +0000 (0:00:00.104) 0:01:27.107 ******* 2021-03-24T20:40:44.963 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-24T20:40:44.964 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:44.990 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:44 +0000 (0:00:00.129) 0:01:27.236 ******* 2021-03-24T20:40:45.070 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-24T20:40:45.072 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:45.101 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:45 +0000 (0:00:00.110) 0:01:27.347 ******* 2021-03-24T20:40:45.181 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-24T20:40:45.183 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:45.210 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:45 +0000 (0:00:00.107) 0:01:27.454 ******* 2021-03-24T20:40:45.323 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-03-24T20:40:45.393 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 24 March 2021 20:40:45 +0000 (0:00:00.143) 0:01:27.598 ******* 2021-03-24T20:40:45.437 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-03-24T20:40:45.462 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:45 +0000 (0:00:00.109) 0:01:27.707 ******* 2021-03-24T20:40:45.542 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-03-24T20:40:45.571 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:45 +0000 (0:00:00.108) 0:01:27.816 ******* 2021-03-24T20:40:45.644 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-03-24T20:40:45.686 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:45 +0000 (0:00:00.115) 0:01:27.932 ******* 2021-03-24T20:40:45.707 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** 2021-03-24T20:40:45.710 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1616618145, "cache_updated": false, "changed": false} 2021-03-24T20:40:45.734 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:45 +0000 (0:00:00.047) 0:01:27.979 ******* 2021-03-24T20:40:45.836 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-03-24T20:40:45.838 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:45.860 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:45 +0000 (0:00:00.126) 0:01:28.106 ******* 2021-03-24T20:40:45.989 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-03-24T20:40:45.998 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi063.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:46.033 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:46 +0000 (0:00:00.172) 0:01:28.279 ******* 2021-03-24T20:40:46.150 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-03-24T20:40:46.153 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:46.175 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:46 +0000 (0:00:00.141) 0:01:28.420 ******* 2021-03-24T20:40:46.311 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "4880e187ff9f437cc33858eb1d0fa49a06d67be4", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "size": 180, "state": "file", "uid": 0} 2021-03-24T20:40:46.339 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:46 +0000 (0:00:00.163) 0:01:28.584 ******* 2021-03-24T20:40:46.347 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T20:40:46.349 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:46.394 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:46 +0000 (0:00:00.049) 0:01:28.634 ******* 2021-03-24T20:40:46.471 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:46.496 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:46 +0000 (0:00:00.108) 0:01:28.742 ******* 2021-03-24T20:40:46.588 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-03-24T20:40:46.590 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:46.625 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:46 +0000 (0:00:00.127) 0:01:28.870 ******* 2021-03-24T20:40:46.697 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-03-24T20:40:46.699 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:46.722 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:46 +0000 (0:00:00.097) 0:01:28.968 ******* 2021-03-24T20:40:46.774 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-03-24T20:40:46.777 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/nagios/plugins/check_mem.sh", "size": 1964, "state": "file", "uid": 0} 2021-03-24T20:40:46.801 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:46 +0000 (0:00:00.078) 0:01:29.047 ******* 2021-03-24T20:40:46.809 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-03-24T20:40:46.811 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:46.840 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:46 +0000 (0:00:00.038) 0:01:29.085 ******* 2021-03-24T20:40:46.922 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-24T20:40:46.924 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:46.947 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:46 +0000 (0:00:00.106) 0:01:29.192 ******* 2021-03-24T20:40:47.027 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-24T20:40:47.029 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "DAEMON_OPTS", "msg": ""} TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:47.054 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:47 +0000 (0:00:00.106) 0:01:29.299 ******* 2021-03-24T20:40:47.125 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-24T20:40:47.126 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:47.151 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:47 +0000 (0:00:00.097) 0:01:29.397 ******* 2021-03-24T20:40:47.159 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-03-24T20:40:47.161 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "backup": "", "changed": false, "item": "NRPE_OPTS", "msg": ""} 2021-03-24T20:40:47.188 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:47 +0000 (0:00:00.037) 0:01:29.434 ******* 2021-03-24T20:40:47.228 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-03-24T20:40:47.253 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:47.257 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:47 +0000 (0:00:00.068) 0:01:29.503 ******* 2021-03-24T20:40:47.275 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-03-24T20:40:47.277 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:47.305 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:47 +0000 (0:00:00.047) 0:01:29.550 ******* 2021-03-24T20:40:47.336 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-24T20:40:47.338 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:47.365 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:47 +0000 (0:00:00.060) 0:01:29.611 ******* 2021-03-24T20:40:47.393 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** 2021-03-24T20:40:47.395 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-03-24T20:40:47.420 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:47 +0000 (0:00:00.054) 0:01:29.665 ******* 2021-03-24T20:40:47.449 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-24T20:40:47.451 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:47.475 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:47 +0000 (0:00:00.055) 0:01:29.721 ******* 2021-03-24T20:40:47.503 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-03-24T20:40:47.505 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:47.529 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:47 +0000 (0:00:00.053) 0:01:29.775 ******* 2021-03-24T20:40:47.569 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-03-24T20:40:47.571 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:47.600 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:47 +0000 (0:00:00.068) 0:01:29.843 ******* 2021-03-24T20:40:47.701 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-03-24T20:40:47.707 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:47.758 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:47 +0000 (0:00:00.160) 0:01:30.004 ******* 2021-03-24T20:40:47.865 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:47.901 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:47 +0000 (0:00:00.143) 0:01:30.147 ******* 2021-03-24T20:40:48.020 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:48.044 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:48 +0000 (0:00:00.142) 0:01:30.289 ******* 2021-03-24T20:40:48.047 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:48.168 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-03-24T20:40:48.172 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:48.199 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:48 +0000 (0:00:00.154) 0:01:30.444 ******* 2021-03-24T20:40:48.206 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-03-24T20:40:48.207 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "a0ee4c31e844cf26108115f6af99f92f904eafbf", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "size": 1176, "state": "file", "uid": 0} 2021-03-24T20:40:48.234 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:48 +0000 (0:00:00.035) 0:01:30.479 ******* 2021-03-24T20:40:48.328 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-03-24T20:40:48.329 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:48.363 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:48 +0000 (0:00:00.128) 0:01:30.608 ******* 2021-03-24T20:40:48.464 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:48.488 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:48 +0000 (0:00:00.125) 0:01:30.733 ******* 2021-03-24T20:40:48.490 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:48.538 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-03-24T20:40:48.539 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nagios-nrpe-server", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-03-24 20:38:44 UTC", "ActiveEnterTimestampMonotonic": "17548837", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "network.target remote-fs.target basic.target -.mount systemd-tmpfiles-setup.service sysinit.target var-run.mount system.slice nss-lookup.target time-sync.target local-fs.target systemd-journald.socket", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-03-24 20:38:44 UTC", "AssertTimestampMonotonic": "17548178", "Before": "plymouth-quit.service xdm.service shutdown.target multi-user.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-03-24 20:38:44 UTC", "ConditionTimestampMonotonic": "17548176", "ConfigurationDirectoryMode": "0755", "Conflicts": "nrpe.socket shutdown.target", "ControlGroup": "/system.slice/nagios-nrpe-ser 2021-03-24T20:40:48.541 INFO:teuthology.task.ansible.out:ver.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EnvironmentFile": "/etc/default/nagios-nrpe-server (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1262", "ExecMainStartTimestamp": "Wed 2021-03-24 20:38:44 UTC", "ExecMainStartTimestampMonotonic": "17548811", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f $NRPE_OPTS ; ignore_errors=no ; start_time=[Wed 2021-03-24 20:38:44 UTC] ; stop_time=[n/a] ; pid=1262 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nagios/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/nagios-nrpe-server.service", "GID": "118", "Group": "nagios", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nagios-nrpe-server.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-03-24 20:38:44 UTC", "InactiveExitTimestampMonotonic": "17548837", "InvocationID": "28aaa0ca08e44e3c92f9c1f7b6295194", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "i 2021-03-24T20:40:48.543 INFO:teuthology.task.ansible.out:nfinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1262", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "nagios-nrpe-server.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nagios/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", 2021-03-24T20:40:48.544 INFO:teuthology.task.ansible.out: "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-03-24 20:38:44 UTC", "StateChangeTimestampMonotonic": "17548837", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "109", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "User": "nagios", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-03-24 20:38:44 UTC", "WatchdogTimestampMonotonic": "17548837", "WatchdogUSec": "0"}} 2021-03-24T20:40:48.564 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:48 +0000 (0:00:00.076) 0:01:30.810 ******* 2021-03-24T20:40:48.584 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-03-24T20:40:48.585 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:48.611 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:48 +0000 (0:00:00.047) 0:01:30.857 ******* 2021-03-24T20:40:48.653 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* 2021-03-24T20:40:48.655 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:48.679 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:48 +0000 (0:00:00.067) 0:01:30.925 ******* 2021-03-24T20:40:48.763 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-03-24T20:40:48.764 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:48.789 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:48 +0000 (0:00:00.109) 0:01:31.034 ******* 2021-03-24T20:40:48.829 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-03-24T20:40:48.831 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.003406", "end": "2021-03-24 20:40:48.400887", "rc": 0, "start": "2021-03-24 20:40:48.397481", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:40:48.860 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:48 +0000 (0:00:00.067) 0:01:31.102 ******* 2021-03-24T20:40:48.875 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-03-24T20:40:48.877 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:48.905 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:48 +0000 (0:00:00.048) 0:01:31.151 ******* 2021-03-24T20:40:48.934 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:48.936 INFO:teuthology.task.ansible.out:TASK [testnode : Remove custom repos] ****************************************** 2021-03-24T20:40:48.971 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:48 +0000 (0:00:00.064) 0:01:31.215 ******* 2021-03-24T20:40:48.993 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-03-24T20:40:48.994 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:49.021 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:49 +0000 (0:00:00.050) 0:01:31.266 ******* 2021-03-24T20:40:49.127 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:49.151 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:49 +0000 (0:00:00.130) 0:01:31.397 ******* 2021-03-24T20:40:49.233 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-03-24T20:40:49.235 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:49.265 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:49 +0000 (0:00:00.113) 0:01:31.510 ******* 2021-03-24T20:40:49.367 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-03-24T20:40:49.370 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:49.400 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:49 +0000 (0:00:00.135) 0:01:31.646 ******* 2021-03-24T20:40:49.414 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** 2021-03-24T20:40:49.416 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-03-24T20:40:49.458 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:49 +0000 (0:00:00.057) 0:01:31.703 ******* 2021-03-24T20:40:49.499 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-03-24T20:40:49.501 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:49.535 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:49 +0000 (0:00:00.077) 0:01:31.781 ******* 2021-03-24T20:40:49.578 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-03-24T20:40:49.580 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:49.627 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:49 +0000 (0:00:00.091) 0:01:31.873 ******* 2021-03-24T20:40:49.653 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:49.699 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:49 +0000 (0:00:00.071) 0:01:31.945 ******* 2021-03-24T20:40:49.829 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-03-24T20:40:49.831 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:49.886 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:49 +0000 (0:00:00.186) 0:01:32.131 ******* 2021-03-24T20:40:50.027 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-03-24T20:40:50.029 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"apt_repos": [], "ceph_packages_to_remove": ["ceph", "ceph-common", "libcephfs1", "radosgw", "python-ceph", "python-rados", "python-cephfs", "python-rbd", "librbd1", "librados2", "ceph-fs-common-dbg", "ceph-fs-common"], "common_apt_repos": [], "common_packages": [], "nfs_service": "nfs-kernel-server", "ntp_service_name": "ntp", "packages": [], "packages_to_remove": ["multipath-tools"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "ssh"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/apt_systems.yml"], "changed": false} 2021-03-24T20:40:50.073 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:50 +0000 (0:00:00.185) 0:01:32.319 ******* 2021-03-24T20:40:50.237 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-03-24T20:40:50.238 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml) => {"ansible_facts": {"common_packages": ["libfcgi0ldbl", "python-dev", "libev-dev", "perl", "libwww-perl", "lsb-release", "build-essential", "sysstat", "gdb", "libedit2", "cryptsetup-bin", "xfsprogs", "gdisk", "parted", "libuuid1", "libatomic-ops-dev", "git-core", "attr", "dbench", "bonnie++", "valgrind", "ant", "libtool", "automake", "gettext", "uuid-dev", "libacl1-dev", "bc", "xfsdump", "xfslibs-dev", "libattr1-dev", "quota", "libcap2-bin", "libncurses5-dev", "lvm2", "vim", "pdsh", "blktrace", "genisoimage", "libjson-xs-perl", "xml-twig-tools", "default-jdk", "junit4", "tgt", "open-iscsi", "cifs-utils", "ipcalc", "nfs-common", "nfs-kernel-server", "software-properties-common"], "no_recommended_packages": ["collectl"], "non_aarch64_common_packages": ["smbios-utils", "libfcgi", "sysprof"], "non_aarch64_packages_to_upgrade": ["libapache2-mod-fastcgi"], "packages_to_remove": ["openmpi-common"], "packages_to_upgrade": ["apt", "apache2"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu.yml"} 2021-03-24T20:40:50.279 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:50 +0000 (0:00:00.206) 0:01:32.525 ******* 2021-03-24T20:40:50.398 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-03-24T20:40:50.400 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml) => {"ansible_facts": {"non_aarch64_packages": ["libgoogle-perftools4", "iozone3", "libssl1.0.0"], "non_aarch64_packages_to_upgrade": [], "packages": ["mpich", "qemu-system-x86", "python-virtualenv", "python-configobj", "python-gevent", "python-numpy", "python-matplotlib", "python-nose", "btrfs-tools", "lttng-tools", "libtool-bin", "docker.io", "python3-nose"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/ubuntu_18.yml"} 2021-03-24T20:40:50.424 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:50 +0000 (0:00:00.144) 0:01:32.669 ******* 2021-03-24T20:40:50.593 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-03-24T20:40:50.595 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-03-24T20:40:50.625 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:50 +0000 (0:00:00.201) 0:01:32.871 ******* 2021-03-24T20:40:50.800 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-03-24T20:40:50.802 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 27, "name": "sudo", "state": "present", "system": false} 2021-03-24T20:40:50.833 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:50 +0000 (0:00:00.207) 0:01:33.079 ******* 2021-03-24T20:40:51.063 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-03-24T20:40:51.064 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "name": "ubuntu", "state": "present", "system": false} 2021-03-24T20:40:51.097 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:51 +0000 (0:00:00.264) 0:01:33.343 ******* 2021-03-24T20:40:51.171 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-03-24T20:40:51.174 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:51.204 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:51 +0000 (0:00:00.106) 0:01:33.450 ******* 2021-03-24T20:40:51.226 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-24T20:40:51.228 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-03-24T20:40:51.294 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:51.323 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:51 +0000 (0:00:00.118) 0:01:33.568 ******* 2021-03-24T20:40:51.570 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-03-24T20:40:51.572 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/sh", "state": "present", "uid": 10101} 2021-03-24T20:40:51.595 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:51 +0000 (0:00:00.272) 0:01:33.841 ******* 2021-03-24T20:40:52.058 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-03-24T20:40:52.060 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "size": 112, "state": "file", "uid": 0} 2021-03-24T20:40:52.089 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:52 +0000 (0:00:00.494) 0:01:34.335 ******* 2021-03-24T20:40:52.184 INFO:teuthology.task.ansible.out: TASK [testnode : Set hostname var] ********************************************* 2021-03-24T20:40:52.191 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi052"}, "changed": false} TASK [testnode : Install apt keys] ********************************************* ok: [smithi063.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-03-24T20:40:52.229 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:52 +0000 (0:00:00.139) 0:01:34.474 ******* 2021-03-24T20:40:52.258 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:52 +0000 (0:00:00.028) 0:01:34.503 ******* 2021-03-24T20:40:52.733 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-03-24T20:40:52.735 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi052.front.sepia.ceph.com", "ansible_hostname": "smithi052", "ansible_nodename": "smithi052"}, "changed": true, "name": "smithi052"} 2021-03-24T20:40:52.760 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:52 +0000 (0:00:00.502) 0:01:35.006 ******* 2021-03-24T20:40:53.161 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-03-24T20:40:53.162 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "2f7bfea2455a0d9220592aa0c21ac69c312e7471", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "size": 2646, "state": "file", "uid": 0} 2021-03-24T20:40:53.191 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:53 +0000 (0:00:00.430) 0:01:35.437 ******* 2021-03-24T20:40:53.283 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-03-24T20:40:53.285 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"cache_update_time": 1616618190, "cache_updated": false, "changed": false} 2021-03-24T20:40:53.311 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:53 +0000 (0:00:00.120) 0:01:35.557 ******* 2021-03-24T20:40:53.384 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-03-24T20:40:53.409 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:53 +0000 (0:00:00.097) 0:01:35.654 ******* 2021-03-24T20:40:53.586 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-03-24T20:40:53.588 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "size": 570, "state": "file", "uid": 0} 2021-03-24T20:40:53.612 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:53 +0000 (0:00:00.203) 0:01:35.858 ******* 2021-03-24T20:40:53.632 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** [WARNING]: Consider using 'become', 'become_method', and 'become_user' rather than running sudo 2021-03-24T20:40:53.634 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.036861", "end": "2021-03-24 20:40:53.204661", "rc": 0, "start": "2021-03-24 20:40:53.167800", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:40:53.665 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:53 +0000 (0:00:00.052) 0:01:35.910 ******* 2021-03-24T20:40:54.109 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-03-24T20:40:54.110 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-03-24T20:40:54.133 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:54 +0000 (0:00:00.467) 0:01:36.378 ******* 2021-03-24T20:40:54.205 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-03-24T20:40:54.207 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:54.231 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:54 +0000 (0:00:00.098) 0:01:36.477 ******* 2021-03-24T20:40:54.302 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-24T20:40:54.304 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:54.327 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:54 +0000 (0:00:00.095) 0:01:36.573 ******* 2021-03-24T20:40:54.398 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-24T20:40:54.400 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:54.429 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:54 +0000 (0:00:00.102) 0:01:36.675 ******* 2021-03-24T20:40:54.510 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-24T20:40:54.511 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:54.534 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:54 +0000 (0:00:00.104) 0:01:36.780 ******* 2021-03-24T20:40:54.606 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-24T20:40:54.608 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:54.638 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:54 +0000 (0:00:00.103) 0:01:36.884 ******* 2021-03-24T20:40:54.718 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-24T20:40:54.720 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:54.746 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:54 +0000 (0:00:00.108) 0:01:36.992 ******* 2021-03-24T20:40:54.816 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-24T20:40:54.817 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:54.839 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:54 +0000 (0:00:00.093) 0:01:37.085 ******* 2021-03-24T20:40:54.921 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-03-24T20:40:54.923 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:54.950 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:54 +0000 (0:00:00.110) 0:01:37.196 ******* 2021-03-24T20:40:55.022 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-03-24T20:40:55.046 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:55 +0000 (0:00:00.095) 0:01:37.291 ******* 2021-03-24T20:40:55.117 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-03-24T20:40:55.142 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:55 +0000 (0:00:00.096) 0:01:37.388 ******* 2021-03-24T20:40:55.212 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-03-24T20:40:55.235 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:55 +0000 (0:00:00.092) 0:01:37.481 ******* 2021-03-24T20:40:55.305 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-03-24T20:40:55.307 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:55.330 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:55 +0000 (0:00:00.095) 0:01:37.576 ******* 2021-03-24T20:40:55.421 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-03-24T20:40:55.423 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-03-24T20:40:55.457 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:55 +0000 (0:00:00.126) 0:01:37.703 ******* 2021-03-24T20:40:55.539 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-03-24T20:40:55.541 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:55.563 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:55 +0000 (0:00:00.106) 0:01:37.809 ******* 2021-03-24T20:40:55.643 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T20:40:55.645 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:55.666 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:55 +0000 (0:00:00.103) 0:01:37.912 ******* 2021-03-24T20:40:55.736 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-03-24T20:40:55.738 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:55.760 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:55 +0000 (0:00:00.093) 0:01:38.006 ******* 2021-03-24T20:40:55.842 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-03-24T20:40:55.843 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:55.866 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:55 +0000 (0:00:00.105) 0:01:38.112 ******* 2021-03-24T20:40:55.936 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-03-24T20:40:55.938 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:55.964 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:55 +0000 (0:00:00.097) 0:01:38.210 ******* 2021-03-24T20:40:56.033 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-03-24T20:40:56.035 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:56.057 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:56 +0000 (0:00:00.092) 0:01:38.303 ******* 2021-03-24T20:40:56.128 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-24T20:40:56.130 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:56.152 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:56 +0000 (0:00:00.094) 0:01:38.397 ******* 2021-03-24T20:40:56.233 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:56.256 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:56 +0000 (0:00:00.104) 0:01:38.502 ******* 2021-03-24T20:40:56.327 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:56.352 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:56 +0000 (0:00:00.095) 0:01:38.598 ******* 2021-03-24T20:40:56.424 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-03-24T20:40:56.426 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:56.447 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:56 +0000 (0:00:00.095) 0:01:38.693 ******* 2021-03-24T20:40:56.517 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:56.541 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:56 +0000 (0:00:00.093) 0:01:38.786 ******* 2021-03-24T20:40:56.611 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-24T20:40:56.613 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:56.636 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:56 +0000 (0:00:00.095) 0:01:38.882 ******* 2021-03-24T20:40:56.718 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-03-24T20:40:56.720 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:56.753 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:56 +0000 (0:00:00.116) 0:01:38.998 ******* 2021-03-24T20:40:56.825 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-03-24T20:40:56.827 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:56.849 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:56 +0000 (0:00:00.096) 0:01:39.094 ******* 2021-03-24T20:40:56.929 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-03-24T20:40:56.930 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:56.953 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:56 +0000 (0:00:00.104) 0:01:39.199 ******* 2021-03-24T20:40:57.023 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-03-24T20:40:57.025 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:57.048 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:57 +0000 (0:00:00.094) 0:01:39.294 ******* 2021-03-24T20:40:57.129 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-03-24T20:40:57.130 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:57.157 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:57 +0000 (0:00:00.108) 0:01:39.402 ******* 2021-03-24T20:40:57.247 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-03-24T20:40:57.249 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:57.270 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:57 +0000 (0:00:00.113) 0:01:39.516 ******* 2021-03-24T20:40:57.339 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-03-24T20:40:57.353 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:57.363 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:57 +0000 (0:00:00.092) 0:01:39.609 ******* 2021-03-24T20:40:57.444 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-03-24T20:40:57.446 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:57.469 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:57 +0000 (0:00:00.105) 0:01:39.714 ******* 2021-03-24T20:40:57.675 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-03-24T20:40:57.677 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": "ls -1 /etc/apt/sources.list.d/", "delta": "0:00:00.005229", "end": "2021-03-24 20:40:57.469684", "rc": 0, "start": "2021-03-24 20:40:57.464455", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:40:57.711 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:57 +0000 (0:00:00.241) 0:01:39.956 ******* 2021-03-24T20:40:57.794 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-03-24T20:40:57.818 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:57 +0000 (0:00:00.107) 0:01:40.064 ******* 2021-03-24T20:40:58.286 INFO:teuthology.task.ansible.out: 2021-03-24T20:40:58.288 INFO:teuthology.task.ansible.out:TASK [testnode : Set apt preferences] ****************************************** ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "ba101482124806c7f6ea9fc7b4acfda303391ad6", "dest": "/etc/apt/preferences.d/ceph.pref", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/apt/preferences.d/ceph.pref", "size": 52, "state": "file", "uid": 0} 2021-03-24T20:40:58.314 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:58 +0000 (0:00:00.495) 0:01:40.559 ******* 2021-03-24T20:40:58.396 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-03-24T20:40:58.398 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:40:58.425 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:58 +0000 (0:00:00.111) 0:01:40.671 ******* 2021-03-24T20:40:59.169 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-03-24T20:40:59.171 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"cache_update_time": 1616618456, "cache_updated": true, "changed": true} 2021-03-24T20:40:59.196 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:40:59 +0000 (0:00:00.770) 0:01:41.442 ******* 2021-03-24T20:40:59.541 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-03-24T20:40:59.543 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc"} 2021-03-24T20:41:00.498 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc"} 2021-03-24T20:41:00.524 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:00 +0000 (0:00:01.327) 0:01:42.769 ******* 2021-03-24T20:41:01.493 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-03-24T20:41:01.495 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1616618145, "cache_updated": false, "changed": false} 2021-03-24T20:41:01.517 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:01 +0000 (0:00:00.993) 0:01:43.762 ******* 2021-03-24T20:41:01.599 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-03-24T20:41:01.623 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:01 +0000 (0:00:00.106) 0:01:43.868 ******* 2021-03-24T20:41:01.873 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-03-24T20:41:01.875 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["sudo", "apt-get", "clean"], "delta": "0:00:00.036643", "end": "2021-03-24 20:41:01.672739", "rc": 0, "start": "2021-03-24 20:41:01.636096", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:01.897 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:01 +0000 (0:00:00.274) 0:01:44.143 ******* 2021-03-24T20:41:07.560 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-03-24T20:41:07.562 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1616618465, "cache_updated": true, "changed": true} 2021-03-24T20:41:07.584 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:07 +0000 (0:00:05.686) 0:01:49.830 ******* 2021-03-24T20:41:09.760 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-03-24T20:41:09.761 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 30 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137596 files and directorie 2021-03-24T20:41:09.763 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 30 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-03-24T20:41:09.765 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137596 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-24T20:41:09.785 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:09 +0000 (0:00:02.201) 0:01:52.031 ******* 2021-03-24T20:41:10.752 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-24T20:41:10.754 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"cache_update_time": 1616618456, "cache_updated": false, "changed": false} 2021-03-24T20:41:10.777 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:10 +0000 (0:00:00.991) 0:01:53.022 ******* 2021-03-24T20:41:10.847 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-03-24T20:41:10.849 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:10.871 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:10 +0000 (0:00:00.094) 0:01:53.117 ******* 2021-03-24T20:41:18.300 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-03-24T20:41:18.302 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following packages were automatically installed and are no longer required:\n cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good\n gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0\n libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1\n libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0\n libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0\n libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3\n libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0\n libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5\n libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils\nUse 'sudo apt autoremove' to remove them.\nThe following packages will be REMOVED:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\n0 upgraded, 0 newly installed, 6 to remove and 30 not upgraded.\nAfter this operation, 47.6 MB disk space will be freed.\n(Reading database ... \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137596 files and directorie 2021-03-24T20:41:18.303 INFO:teuthology.task.ansible.out:s currently installed.)\r\nRemoving qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nRemoving librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nRemoving librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following packages were automatically installed and are no longer required:", " cpu-checker gstreamer1.0-plugins-base gstreamer1.0-plugins-good", " gstreamer1.0-x ipxe-qemu ipxe-qemu-256k-compat-efi-roms libaa1 libavc1394-0", " libbluetooth3 libbrlapi0.6 libcacard0 libcdparanoia0 libdv4 libfdt1", " libgstreamer-plugins-base1.0-0 libgstreamer-plugins-good1.0-0", " libgstreamer1.0-0 libiec61883-0 libiscsi7 libjack-jackd2-0 libmp3lame0", " libmpg123-0 libopus0 liborc-0.4-0 libraw1394-11 libsamplerate0 libshout3", " libspeex1 libspice-server1 libtag1v5 libtag1v5-vanilla libtwolame0", " libusbredirparser1 libv4l-0 libv4lconvert0 libvisual-0.4-0 libvpx5", " libwavpack1 libxen-4.9 libxenstore3.0 libyajl2 msr-tools seabios sharutils", "Use 'sudo apt autoremove' to remove them.", "The following packages will be REMOVED:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "0 upgraded, 0 newly installed, 6 to remove and 30 not upgraded.", "After this operation, 47.6 MB disk space will be freed.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Read 2021-03-24T20:41:18.305 INFO:teuthology.task.ansible.out:ing database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137596 files and directories currently installed.)", "Removing qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Removing qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Removing librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Removing librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-24T20:41:18.336 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:18 +0000 (0:00:07.465) 0:02:00.582 ******* 2021-03-24T20:41:19.323 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-24T20:41:19.325 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1616618465, "cache_updated": false, "changed": false} 2021-03-24T20:41:19.347 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:19 +0000 (0:00:01.010) 0:02:01.592 ******* 2021-03-24T20:41:19.420 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-03-24T20:41:19.422 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:19.447 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:19 +0000 (0:00:00.100) 0:02:01.693 ******* 2021-03-24T20:41:21.268 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-24T20:41:21.270 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"cache_update_time": 1616618456, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk\n gitweb git-cvs git-mediawiki git-svn samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\nThe following packages will be upgraded:\n git\n1 upgraded, 6 newly installed, 0 to remove and 29 not upgraded.\nNeed to get 14.3 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.8 [3916 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]\nGet:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]\nFetched 14.3 MB in 1s (11.9 MB/s)\nSelecting previously unselected package librados2.\r\n(Reading database .. 2021-03-24T20:41:21.271 INFO:teuthology.task.ansible.out:. \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137454 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nPreparing to unpack .../4-git_1%3a2.17.1-1ubuntu0.8_amd64.deb ...\r\nUnpacking git (1:2.17.1-1ubuntu0.8) over (1:2.17.1-1ubuntu0.7) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../5-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../6-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up git (1:2.17.1-1ubuntu0.8) ...\r\nSetti 2021-03-24T20:41:21.273 INFO:teuthology.task.ansible.out:ng up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk", " gitweb git-cvs git-mediawiki git-svn samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "The following packages will be upgraded:", " git", "1 upgraded, 6 newly installed, 0 to remove and 29 not upgraded.", "Need to get 14.3 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.8 [3916 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]", "Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.1 2021-03-24T20:41:21.275 INFO:teuthology.task.ansible.out:1+dfsg-1ubuntu7.36 [870 kB]", "Fetched 14.3 MB in 1s (11.9 MB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137454 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Preparing to unpack .../4-git_1%3a2.17.1-1ubuntu0.8_amd64.deb ...", "Unpacking git (1:2.17.1-1ubuntu0.8) over (1:2.17.1-1ubuntu0.7) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../5-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../6-qemu-utils_1%3a2.11+dfsg-1ub 2021-03-24T20:41:21.276 INFO:teuthology.task.ansible.out:untu7.36_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up git (1:2.17.1-1ubuntu0.8) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-24T20:41:21.294 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:21 +0000 (0:00:01.846) 0:02:03.540 ******* 2021-03-24T20:41:22.275 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-03-24T20:41:22.277 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"cache_update_time": 1616618456, "cache_updated": false, "changed": false} 2021-03-24T20:41:22.298 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:22 +0000 (0:00:01.004) 0:02:04.544 ******* 2021-03-24T20:41:23.247 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-03-24T20:41:23.249 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"cache_update_time": 1616618456, "cache_updated": false, "changed": false} 2021-03-24T20:41:23.274 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:23 +0000 (0:00:00.974) 0:02:05.519 ******* 2021-03-24T20:41:23.459 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-24T20:41:23.461 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-03-24T20:41:23.486 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:23 +0000 (0:00:00.212) 0:02:05.731 ******* 2021-03-24T20:41:23.565 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-03-24T20:41:23.567 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:23.593 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:23 +0000 (0:00:00.107) 0:02:05.838 ******* 2021-03-24T20:41:23.778 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-24T20:41:23.780 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-03-24T20:41:23.802 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:23 +0000 (0:00:00.209) 0:02:06.048 ******* 2021-03-24T20:41:24.201 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-03-24T20:41:24.203 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-03-24T20:41:24.227 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:24 +0000 (0:00:00.424) 0:02:06.472 ******* 2021-03-24T20:41:24.446 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-03-24T20:41:24.448 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-03-24T20:41:24.477 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:24 +0000 (0:00:00.249) 0:02:06.722 ******* 2021-03-24T20:41:24.663 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-03-24T20:41:24.666 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T20:41:24.688 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:24 +0000 (0:00:00.211) 0:02:06.934 ******* 2021-03-24T20:41:25.046 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-03-24T20:41:25.047 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Wed 2021-03-24 20:39:07 UTC", "ActiveEnterTimestampMonotonic": "18962016", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "-.mount basic.target nss-lookup.target network.target systemd-journald.socket systemd-tmpfiles-setup.service sysinit.target remote-fs.target system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-03-24 20:39:07 UTC", "AssertTimestampMonotonic": "18578468", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-03-24 20:39:07 UTC", "ConditionTimestampMonotonic": "18578467", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-03-24T20:41:25.049 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1452", "ExecMainStartTimestamp": "Wed 2021-03-24 20:39:07 UTC", "ExecMainStartTimestampMonotonic": "18962004", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-03-24 20:39:07 UTC", "InactiveExitTimestampMonotonic": "18579102", "InvocationID": "203fd13eeef7434595314bc2cdfb0eca", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-03-24T20:41:25.050 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1452", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-03-24T20:41:25.052 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-03-24 20:39:07 UTC", "StateChangeTimestampMonotonic": "18962016", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-03-24 20:39:07 UTC", "WatchdogTimestampMonotonic": "18962014", "WatchdogUSec": "0"}} 2021-03-24T20:41:25.072 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:25 +0000 (0:00:00.383) 0:02:07.318 ******* 2021-03-24T20:41:25.152 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-24T20:41:25.154 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:25.176 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:25 +0000 (0:00:00.103) 0:02:07.422 ******* 2021-03-24T20:41:25.247 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-24T20:41:25.249 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:25.271 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:25 +0000 (0:00:00.095) 0:02:07.517 ******* 2021-03-24T20:41:25.362 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-24T20:41:25.363 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:25.386 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:25 +0000 (0:00:00.114) 0:02:07.631 ******* 2021-03-24T20:41:25.479 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-24T20:41:25.481 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:25.507 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:25 +0000 (0:00:00.121) 0:02:07.753 ******* 2021-03-24T20:41:25.587 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:25.611 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:25 +0000 (0:00:00.104) 0:02:07.857 ******* 2021-03-24T20:41:25.683 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-24T20:41:25.684 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:25.706 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:25 +0000 (0:00:00.094) 0:02:07.952 ******* 2021-03-24T20:41:25.787 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-24T20:41:25.789 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:25.810 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:25 +0000 (0:00:00.104) 0:02:08.056 ******* 2021-03-24T20:41:25.881 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-24T20:41:25.882 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:25.909 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:25 +0000 (0:00:00.098) 0:02:08.154 ******* 2021-03-24T20:41:25.988 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T20:41:25.990 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:26.012 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:26 +0000 (0:00:00.102) 0:02:08.257 ******* 2021-03-24T20:41:26.082 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:26.107 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:26 +0000 (0:00:00.095) 0:02:08.353 ******* 2021-03-24T20:41:26.177 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-24T20:41:26.178 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:26.203 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:26 +0000 (0:00:00.095) 0:02:08.448 ******* 2021-03-24T20:41:26.273 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-24T20:41:26.275 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:26.298 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:26 +0000 (0:00:00.095) 0:02:08.544 ******* 2021-03-24T20:41:26.379 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-24T20:41:26.381 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:26.403 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:26 +0000 (0:00:00.105) 0:02:08.649 ******* 2021-03-24T20:41:26.485 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-24T20:41:26.487 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:26.511 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:26 +0000 (0:00:00.107) 0:02:08.756 ******* 2021-03-24T20:41:26.580 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-24T20:41:26.582 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:26.603 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:26 +0000 (0:00:00.092) 0:02:08.849 ******* 2021-03-24T20:41:26.685 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-24T20:41:26.686 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:26.708 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:26 +0000 (0:00:00.105) 0:02:08.954 ******* 2021-03-24T20:41:26.789 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-24T20:41:26.791 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:26.812 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:26 +0000 (0:00:00.103) 0:02:09.058 ******* 2021-03-24T20:41:26.883 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-24T20:41:26.885 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:26.908 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:26 +0000 (0:00:00.095) 0:02:09.153 ******* 2021-03-24T20:41:26.977 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-24T20:41:26.979 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:27.003 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:27 +0000 (0:00:00.094) 0:02:09.248 ******* 2021-03-24T20:41:27.083 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:27.110 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:27 +0000 (0:00:00.107) 0:02:09.355 ******* 2021-03-24T20:41:27.181 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-24T20:41:27.184 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:27.207 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:27 +0000 (0:00:00.096) 0:02:09.452 ******* 2021-03-24T20:41:27.278 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-24T20:41:27.280 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:27.303 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:27 +0000 (0:00:00.097) 0:02:09.549 ******* 2021-03-24T20:41:27.374 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-24T20:41:27.376 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:27.407 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:27 +0000 (0:00:00.103) 0:02:09.653 ******* 2021-03-24T20:41:27.489 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-24T20:41:27.491 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:27.516 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:27 +0000 (0:00:00.108) 0:02:09.761 ******* 2021-03-24T20:41:27.598 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-24T20:41:27.601 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:27.625 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:27 +0000 (0:00:00.109) 0:02:09.870 ******* 2021-03-24T20:41:27.707 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:27.733 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:27 +0000 (0:00:00.107) 0:02:09.978 ******* 2021-03-24T20:41:27.947 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-03-24T20:41:27.949 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-03-24T20:41:27.975 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:27 +0000 (0:00:00.241) 0:02:10.220 ******* 2021-03-24T20:41:28.399 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-24T20:41:28.401 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-03-24T20:41:28.423 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:28 +0000 (0:00:00.447) 0:02:10.668 ******* 2021-03-24T20:41:28.506 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-24T20:41:28.508 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:28.538 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:28 +0000 (0:00:00.115) 0:02:10.783 ******* 2021-03-24T20:41:28.966 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-03-24T20:41:28.968 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-03-24T20:41:28.994 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:28 +0000 (0:00:00.456) 0:02:11.239 ******* 2021-03-24T20:41:29.436 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-03-24T20:41:29.438 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-03-24T20:41:29.464 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:29 +0000 (0:00:00.470) 0:02:11.710 ******* 2021-03-24T20:41:29.841 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-03-24T20:41:29.843 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-03-24T20:41:29.874 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:29 +0000 (0:00:00.409) 0:02:12.120 ******* 2021-03-24T20:41:29.957 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-03-24T20:41:29.958 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1616618465, "cache_updated": false, "changed": true, "stderr": "W: --force-yes is deprecated, use one of the options starting with --allow instead.\n", "stderr_lines": ["W: --force-yes is deprecated, use one of the options starting with --allow instead."], "stdout": "Reading package lists...\nBuilding dependency tree...\nReading state information...\nThe following additional packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils\nSuggested packages:\n git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk\n gitweb git-cvs git-mediawiki git-svn samba vde2 sgabios ovmf debootstrap\nThe following NEW packages will be installed:\n librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86\n qemu-utils\nThe following packages will be upgraded:\n git\n1 upgraded, 6 newly installed, 0 to remove and 29 not upgraded.\nNeed to get 14.3 MB of archives.\nAfter this operation, 47.6 MB of additional disk space will be used.\nGet:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]\nGet:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]\nGet:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]\nGet:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]\nGet:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.8 [3916 kB]\nGet:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]\nGet:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.11+dfsg-1ubuntu7.36 [870 kB]\nFetched 14.3 MB in 1s (11.9 MB/s)\nSelecting previously unselected package librados2.\r\n(Reading database .. 2021-03-24T20:41:29.961 INFO:teuthology.task.ansible.out:. \r(Reading database ... 5%\r(Reading database ... 10%\r(Reading database ... 15%\r(Reading database ... 20%\r(Reading database ... 25%\r(Reading database ... 30%\r(Reading database ... 35%\r(Reading database ... 40%\r(Reading database ... 45%\r(Reading database ... 50%\r(Reading database ... 55%\r(Reading database ... 60%\r(Reading database ... 65%\r(Reading database ... 70%\r(Reading database ... 75%\r(Reading database ... 80%\r(Reading database ... 85%\r(Reading database ... 90%\r(Reading database ... 95%\r(Reading database ... 100%\r(Reading database ... 137454 files and directories currently installed.)\r\nPreparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package librbd1.\r\nPreparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...\r\nUnpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSelecting previously unselected package qemu-block-extra:amd64.\r\nPreparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-system-common.\r\nPreparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nPreparing to unpack .../4-git_1%3a2.17.1-1ubuntu0.8_amd64.deb ...\r\nUnpacking git (1:2.17.1-1ubuntu0.8) over (1:2.17.1-1ubuntu0.7) ...\r\nSelecting previously unselected package qemu-system-x86.\r\nPreparing to unpack .../5-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSelecting previously unselected package qemu-utils.\r\nPreparing to unpack .../6-qemu-utils_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...\r\nUnpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up librados2 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up git (1:2.17.1-1ubuntu0.8) ...\r\nSetti 2021-03-24T20:41:29.963 INFO:teuthology.task.ansible.out:ng up librbd1 (12.2.13-0ubuntu0.18.04.6) ...\r\nSetting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...\r\nSetting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...\r\nProcessing triggers for man-db (2.8.3-2ubuntu0.1) ...\r\nProcessing triggers for libc-bin (2.27-3ubuntu1.4) ...\r\n", "stdout_lines": ["Reading package lists...", "Building dependency tree...", "Reading state information...", "The following additional packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-utils", "Suggested packages:", " git-daemon-run | git-daemon-sysvinit git-doc git-el git-email git-gui gitk", " gitweb git-cvs git-mediawiki git-svn samba vde2 sgabios ovmf debootstrap", "The following NEW packages will be installed:", " librados2 librbd1 qemu-block-extra qemu-system-common qemu-system-x86", " qemu-utils", "The following packages will be upgraded:", " git", "1 upgraded, 6 newly installed, 0 to remove and 29 not upgraded.", "Need to get 14.3 MB of archives.", "After this operation, 47.6 MB of additional disk space will be used.", "Get:1 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librados2 amd64 12.2.13-0ubuntu0.18.04.6 [2724 kB]", "Get:2 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 librbd1 amd64 12.2.13-0ubuntu0.18.04.6 [925 kB]", "Get:3 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-block-extra amd64 1:2.11+dfsg-1ubuntu7.36 [40.5 kB]", "Get:4 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-common amd64 1:2.11+dfsg-1ubuntu7.36 [672 kB]", "Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 git amd64 1:2.17.1-1ubuntu0.8 [3916 kB]", "Get:6 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-system-x86 amd64 1:2.11+dfsg-1ubuntu7.36 [5176 kB]", "Get:7 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 qemu-utils amd64 1:2.1 2021-03-24T20:41:29.964 INFO:teuthology.task.ansible.out:1+dfsg-1ubuntu7.36 [870 kB]", "Fetched 14.3 MB in 1s (11.9 MB/s)", "Selecting previously unselected package librados2.", "(Reading database ... ", "(Reading database ... 5%", "(Reading database ... 10%", "(Reading database ... 15%", "(Reading database ... 20%", "(Reading database ... 25%", "(Reading database ... 30%", "(Reading database ... 35%", "(Reading database ... 40%", "(Reading database ... 45%", "(Reading database ... 50%", "(Reading database ... 55%", "(Reading database ... 60%", "(Reading database ... 65%", "(Reading database ... 70%", "(Reading database ... 75%", "(Reading database ... 80%", "(Reading database ... 85%", "(Reading database ... 90%", "(Reading database ... 95%", "(Reading database ... 100%", "(Reading database ... 137454 files and directories currently installed.)", "Preparing to unpack .../0-librados2_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package librbd1.", "Preparing to unpack .../1-librbd1_12.2.13-0ubuntu0.18.04.6_amd64.deb ...", "Unpacking librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Selecting previously unselected package qemu-block-extra:amd64.", "Preparing to unpack .../2-qemu-block-extra_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-system-common.", "Preparing to unpack .../3-qemu-system-common_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Preparing to unpack .../4-git_1%3a2.17.1-1ubuntu0.8_amd64.deb ...", "Unpacking git (1:2.17.1-1ubuntu0.8) over (1:2.17.1-1ubuntu0.7) ...", "Selecting previously unselected package qemu-system-x86.", "Preparing to unpack .../5-qemu-system-x86_1%3a2.11+dfsg-1ubuntu7.36_amd64.deb ...", "Unpacking qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Selecting previously unselected package qemu-utils.", "Preparing to unpack .../6-qemu-utils_1%3a2.11+dfsg-1ub 2021-03-24T20:41:29.966 INFO:teuthology.task.ansible.out:untu7.36_amd64.deb ...", "Unpacking qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up librados2 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up git (1:2.17.1-1ubuntu0.8) ...", "Setting up librbd1 (12.2.13-0ubuntu0.18.04.6) ...", "Setting up qemu-block-extra:amd64 (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-utils (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-common (1:2.11+dfsg-1ubuntu7.36) ...", "Setting up qemu-system-x86 (1:2.11+dfsg-1ubuntu7.36) ...", "Processing triggers for man-db (2.8.3-2ubuntu0.1) ...", "Processing triggers for libc-bin (2.27-3ubuntu1.4) ..."]} 2021-03-24T20:41:29.984 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:29 +0000 (0:00:00.110) 0:02:12.230 ******* 2021-03-24T20:41:30.057 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** ok: [smithi063.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T20:41:30.083 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:30 +0000 (0:00:00.098) 0:02:12.329 ******* 2021-03-24T20:41:30.163 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-03-24T20:41:30.165 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:30.187 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:30 +0000 (0:00:00.104) 0:02:12.433 ******* 2021-03-24T20:41:30.259 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-03-24T20:41:30.261 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:30.283 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:30 +0000 (0:00:00.095) 0:02:12.528 ******* 2021-03-24T20:41:30.353 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-03-24T20:41:30.355 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:30.376 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:30 +0000 (0:00:00.093) 0:02:12.622 ******* 2021-03-24T20:41:30.445 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-03-24T20:41:30.447 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:30.472 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:30 +0000 (0:00:00.095) 0:02:12.718 ******* 2021-03-24T20:41:30.552 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-24T20:41:30.554 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:30.577 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:30 +0000 (0:00:00.105) 0:02:12.823 ******* 2021-03-24T20:41:30.659 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T20:41:30.660 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-24T20:41:30.691 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:30 +0000 (0:00:00.113) 0:02:12.936 ******* 2021-03-24T20:41:30.783 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-24T20:41:30.785 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} skipping: [smithi063.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:30.796 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:30.798 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:30.809 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:30.810 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:30.821 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-03-24T20:41:30.832 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:30.833 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:30.845 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:30.846 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} skipping: [smithi063.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:30.857 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:30.859 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:30.870 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:30.899 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:30 +0000 (0:00:00.208) 0:02:13.145 ******* 2021-03-24T20:41:30.958 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-03-24T20:41:30.960 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1616618465, "cache_updated": false, "changed": false} 2021-03-24T20:41:30.988 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:30 +0000 (0:00:00.089) 0:02:13.234 ******* 2021-03-24T20:41:31.145 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-03-24T20:41:31.147 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003202", "end": "2021-03-24 20:41:31.116368", "rc": 0, "start": "2021-03-24 20:41:31.113166", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/virtualenv/bin/teuthology-lock"]} 2021-03-24T20:41:31.173 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:31 +0000 (0:00:00.184) 0:02:13.419 ******* 2021-03-24T20:41:31.256 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-03-24T20:41:31.257 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:31.282 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:31 +0000 (0:00:00.108) 0:02:13.528 ******* 2021-03-24T20:41:31.362 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-03-24T20:41:31.365 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:31.387 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:31 +0000 (0:00:00.104) 0:02:13.632 ******* 2021-03-24T20:41:31.478 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-03-24T20:41:31.480 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:31.503 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:31 +0000 (0:00:00.117) 0:02:13.749 ******* 2021-03-24T20:41:31.928 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages with --no-install-recommends] **************** 2021-03-24T20:41:31.930 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1616618465, "cache_updated": false, "changed": false} 2021-03-24T20:41:31.956 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:31 +0000 (0:00:00.452) 0:02:14.202 ******* 2021-03-24T20:41:32.145 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-03-24T20:41:32.146 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": ["perl", "-pe", "if (m{^([^#]\\S*\\s+/\\s+\\S+\\s+)(\\S+)(\\s+.*)$}) { $_=\"$1$2,user_xattr$3\\n\" unless $2=~m{(^|,)user_xattr(,|$)}; }", "-i.bak", "/etc/fstab"], "rc": 0, "stdout": "skipped, since /etc/fstab.bak exists", "stdout_lines": ["skipped, since /etc/fstab.bak exists"]} 2021-03-24T20:41:32.169 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:32 +0000 (0:00:00.212) 0:02:14.414 ******* 2021-03-24T20:41:32.250 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-03-24T20:41:32.252 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:32.274 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:32 +0000 (0:00:00.105) 0:02:14.519 ******* 2021-03-24T20:41:32.460 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-03-24T20:41:32.462 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 10102, "name": "fuse", "state": "present", "system": false} 2021-03-24T20:41:32.490 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:32 +0000 (0:00:00.216) 0:02:14.736 ******* 2021-03-24T20:41:32.508 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-03-24T20:41:32.510 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"cache_update_time": 1616618456, "cache_updated": false, "changed": false} 2021-03-24T20:41:32.543 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:32 +0000 (0:00:00.052) 0:02:14.789 ******* 2021-03-24T20:41:32.545 INFO:teuthology.task.ansible.out: 2021-03-24T20:41:32.627 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-24T20:41:32.629 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:32.653 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:32 +0000 (0:00:00.110) 0:02:14.899 ******* 2021-03-24T20:41:32.732 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-24T20:41:32.734 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item={'block_used': 13753810, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341298, 'size_available': 927093956608, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149511, 'block_size': 4096, 'inode_available': 60905465}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341298, "block_size": 4096, "block_total": 240095108, "block_used": 13753810, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905465, "inode_total": 61054976, "inode_used": 149511, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927093956608, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}} 2021-03-24T20:41:32.758 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:32 +0000 (0:00:00.105) 0:02:15.004 ******* 2021-03-24T20:41:32.892 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** 2021-03-24T20:41:32.893 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "9edff47031d0f14a9b8c813b44eb4a9296d0439f", "dest": "/etc/fuse.conf", "gid": 10102, "group": "fuse", "mode": "0644", "owner": "root", "path": "/etc/fuse.conf", "size": 366, "state": "file", "uid": 0} 2021-03-24T20:41:32.919 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:32 +0000 (0:00:00.160) 0:02:15.164 ******* 2021-03-24T20:41:32.949 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-24T20:41:32.950 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.011406", "end": "2021-03-24 20:41:32.541212", "rc": 0, "start": "2021-03-24 20:41:32.529806", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:32.975 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:32 +0000 (0:00:00.055) 0:02:15.220 ******* 2021-03-24T20:41:33.056 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-03-24T20:41:33.058 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'block_used': 13753810, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341298, 'size_available': 927093956608, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149511, 'block_size': 4096, 'inode_available': 60905465}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341298, "block_size": 4096, "block_total": 240095108, "block_used": 13753810, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905465, "inode_total": 61054976, "inode_used": 149511, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927093956608, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:33.080 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:33 +0000 (0:00:00.105) 0:02:15.326 ******* 2021-03-24T20:41:33.150 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ 2021-03-24T20:41:33.152 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1238, "groups": "fuse,kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-03-24T20:41:33.175 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:33 +0000 (0:00:00.094) 0:02:15.421 ******* 2021-03-24T20:41:33.197 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:33 +0000 (0:00:00.021) 0:02:15.443 ******* 2021-03-24T20:41:33.288 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-24T20:41:33.290 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:33.373 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-03-24T20:41:33.375 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T20:41:33.397 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:33 +0000 (0:00:00.200) 0:02:15.643 ******* 2021-03-24T20:41:33.752 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-03-24T20:41:33.753 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "name": "apache2", "state": "stopped", "status": {"ActiveEnterTimestamp": "Wed 2021-03-24 20:38:44 UTC", "ActiveEnterTimestampMonotonic": "17941494", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice sysinit.target -.mount basic.target systemd-journald.socket network.target remote-fs.target nss-lookup.target systemd-tmpfiles-setup.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-03-24 20:38:44 UTC", "AssertTimestampMonotonic": "17547459", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-03-24 20:38:44 UTC", "ConditionTimestampMonotonic": "17547459", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ControlGroup": "/system.slice/apache2.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "The Apache HTTP Server", "DevicePolicy": "auto", " 2021-03-24T20:41:33.755 INFO:teuthology.task.ansible.out:DropInPaths": "/lib/systemd/system/apache2.service.d/apache2-systemd.conf", "DynamicUser": "no", "Environment": "APACHE_STARTED_BY_SYSTEMD=true", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1338", "ExecMainStartTimestamp": "Wed 2021-03-24 20:38:44 UTC", "ExecMainStartTimestampMonotonic": "17941483", "ExecMainStatus": "0", "ExecReload": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl graceful ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl start ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStop": "{ path=/usr/sbin/apachectl ; argv[]=/usr/sbin/apachectl stop ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/apache2.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "apache2.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-03-24 20:38:44 UTC", "InactiveExitTimestampMonotonic": "17548127", "InvocationID": "e9544201418244b9a82da3e5ce4ef4e6", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESof 2021-03-24T20:41:33.757 INFO:teuthology.task.ansible.out:t": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1338", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "apache2.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice -.mount", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIG 2021-03-24T20:41:33.758 INFO:teuthology.task.ansible.out:HUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-03-24 20:38:44 UTC", "StateChangeTimestampMonotonic": "17941494", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "55", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-03-24 20:38:44 UTC", "WatchdogTimestampMonotonic": "17941493", "WatchdogUSec": "0"}} 2021-03-24T20:41:33.778 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:33 +0000 (0:00:00.380) 0:02:16.024 ******* 2021-03-24T20:41:33.848 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-03-24T20:41:33.849 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:33.872 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:33 +0000 (0:00:00.093) 0:02:16.117 ******* 2021-03-24T20:41:33.953 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-03-24T20:41:33.954 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:33.980 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:33 +0000 (0:00:00.108) 0:02:16.226 ******* 2021-03-24T20:41:34.050 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-03-24T20:41:34.053 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:34.074 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:34 +0000 (0:00:00.093) 0:02:16.319 ******* 2021-03-24T20:41:34.143 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-03-24T20:41:34.145 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:34.169 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:34 +0000 (0:00:00.095) 0:02:16.414 ******* 2021-03-24T20:41:34.238 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-03-24T20:41:34.240 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:34.262 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:34 +0000 (0:00:00.092) 0:02:16.507 ******* 2021-03-24T20:41:34.343 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-03-24T20:41:34.344 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:34.370 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:34 +0000 (0:00:00.108) 0:02:16.615 ******* 2021-03-24T20:41:34.441 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-03-24T20:41:34.443 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:34.465 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:34 +0000 (0:00:00.094) 0:02:16.710 ******* 2021-03-24T20:41:34.539 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-03-24T20:41:34.540 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:34.565 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:34 +0000 (0:00:00.100) 0:02:16.810 ******* 2021-03-24T20:41:34.634 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T20:41:34.636 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:34.658 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:34 +0000 (0:00:00.093) 0:02:16.904 ******* 2021-03-24T20:41:34.738 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-03-24T20:41:34.740 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:34.764 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:34 +0000 (0:00:00.105) 0:02:17.009 ******* 2021-03-24T20:41:34.855 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:34.877 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:34 +0000 (0:00:00.113) 0:02:17.123 ******* 2021-03-24T20:41:34.948 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-03-24T20:41:34.950 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:34.972 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:34 +0000 (0:00:00.094) 0:02:17.217 ******* 2021-03-24T20:41:35.052 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:35.075 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:35 +0000 (0:00:00.103) 0:02:17.321 ******* 2021-03-24T20:41:35.145 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-24T20:41:35.147 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:35.172 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:35 +0000 (0:00:00.096) 0:02:17.418 ******* 2021-03-24T20:41:35.242 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-24T20:41:35.244 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:35.266 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:35 +0000 (0:00:00.093) 0:02:17.512 ******* 2021-03-24T20:41:35.285 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-24T20:41:35.286 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HTevsRnLTJgDpfoJoI4wwlEpqcIi58LFYt'], 'uuids': ['c31e5e1d-e590-45e2-a0c6-ed280195ce53']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.869760", "end": "2021-03-24 20:41:34.872179", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HTevsRnLTJgDpfoJoI4wwlEpqcIi58LFYt"], "labels": [], "masters": [], "uuids": ["c31e5e1d-e590-45e2-a0c6-ed280195ce53"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:33.002419", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T20:41:35.298 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:35.299 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:35.342 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-24T20:41:35.344 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:35.366 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:35 +0000 (0:00:00.099) 0:02:17.611 ******* 2021-03-24T20:41:35.435 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-24T20:41:35.437 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:35.460 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:35 +0000 (0:00:00.094) 0:02:17.705 ******* 2021-03-24T20:41:35.541 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-24T20:41:35.543 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:35.565 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:35 +0000 (0:00:00.105) 0:02:17.811 ******* 2021-03-24T20:41:35.637 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-03-24T20:41:35.638 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:35.674 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:35 +0000 (0:00:00.108) 0:02:17.919 ******* 2021-03-24T20:41:35.747 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-24T20:41:35.749 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:35.770 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:35 +0000 (0:00:00.096) 0:02:18.016 ******* 2021-03-24T20:41:35.840 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-24T20:41:35.843 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:35.864 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:35 +0000 (0:00:00.093) 0:02:18.110 ******* 2021-03-24T20:41:35.934 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-24T20:41:35.936 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:35.958 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:35 +0000 (0:00:00.093) 0:02:18.204 ******* 2021-03-24T20:41:36.038 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-03-24T20:41:36.040 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:36.063 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:36 +0000 (0:00:00.104) 0:02:18.308 ******* 2021-03-24T20:41:36.153 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-03-24T20:41:36.155 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:36.177 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:36 +0000 (0:00:00.114) 0:02:18.423 ******* 2021-03-24T20:41:36.252 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-03-24T20:41:36.254 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:36.275 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:36 +0000 (0:00:00.098) 0:02:18.521 ******* 2021-03-24T20:41:36.346 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-03-24T20:41:36.348 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:36.375 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:36 +0000 (0:00:00.099) 0:02:18.621 ******* 2021-03-24T20:41:36.550 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-24T20:41:36.552 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HTcM7900FBnGilVFWJ9pySp92ZdLH3ddjC'], 'uuids': ['dc6551df-c4bf-4f5b-bd97-06312aa04002']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.119090", "end": "2021-03-24 20:41:36.138327", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HTcM7900FBnGilVFWJ9pySp92ZdLH3ddjC"], "labels": [], "masters": [], "uuids": ["dc6551df-c4bf-4f5b-bd97-06312aa04002"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:35.019237", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T20:41:36.563 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W2X1', 'wwn-0x5000c50087b8ff21'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W2X1-part1', 'wwn-0x5000c50087b8ff21-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50087b8ff21', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4W2X1', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W2X1", "wwn-0x5000c50087b8ff21"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W2X1-part1", "wwn-0x5000c50087b8ff21-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4W2X1", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50087b8ff21"}}, "skip_reason": 2021-03-24T20:41:36.564 INFO:teuthology.task.ansible.out:"Conditional result was False"} TASK [testnode : Remove /etc/ceph] ********************************************* ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "path": "/etc/ceph", "state": "absent"} 2021-03-24T20:41:36.589 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:36 +0000 (0:00:00.213) 0:02:18.834 ******* 2021-03-24T20:41:36.974 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-03-24T20:41:36.976 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "0b949a4622a315acfc90571fddbd013b4dc0f5f9", "dest": "/etc/exports", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/exports", "size": 578, "state": "file", "uid": 0} 2021-03-24T20:41:36.998 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:36 +0000 (0:00:00.409) 0:02:19.243 ******* 2021-03-24T20:41:37.067 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-03-24T20:41:37.069 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:37.092 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:37 +0000 (0:00:00.094) 0:02:19.338 ******* 2021-03-24T20:41:37.594 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-03-24T20:41:37.596 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "d5677bbd13857966eb0fc86f66195b18a2b98e20", "dest": "/etc/cron.weekly/kernel-clean", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/cron.weekly/kernel-clean", "size": 752, "state": "file", "uid": 0} 2021-03-24T20:41:37.619 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:37 +0000 (0:00:00.527) 0:02:19.865 ******* 2021-03-24T20:41:37.829 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-24T20:41:37.831 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-nMdmyw-mycu-nbWw-1O9F-r2Ag-7c6m-iD0Ji9', 'nvme-INTEL_SSDPEDMD400G4_CVFT5344005B400BGN', 'nvme-nvme.8086-43564654353334343030354234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.141993", "end": "2021-03-24 20:41:37.416447", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-nMdmyw-mycu-nbWw-1O9F-r2Ag-7c6m-iD0Ji9", "nvme-INTEL_SSDPEDMD400G4_CVFT5344005B400BGN", "nvme-nvme.8086-43564654353334343030354234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:36.274454", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-03-24T20:41:37.833 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T20:41:37.841 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:37.853 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:38.041 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-03-24T20:41:38.043 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "22e9a1446f43d0440219b9af14a66dd77fb0c2eb", "dest": "/etc/grub.d/02_force_timeout", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/grub.d/02_force_timeout", "size": 115, "state": "file", "uid": 0} 2021-03-24T20:41:38.064 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:38 +0000 (0:00:00.445) 0:02:20.310 ******* 2021-03-24T20:41:38.438 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-03-24T20:41:38.440 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "6449466d762bced58744df80432d189e35329803", "dest": "/etc/modules", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/modules", "size": 315, "state": "file", "uid": 0} 2021-03-24T20:41:38.462 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:38 +0000 (0:00:00.397) 0:02:20.708 ******* 2021-03-24T20:41:38.648 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-03-24T20:41:38.650 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T20:41:38.674 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:38 +0000 (0:00:00.211) 0:02:20.920 ******* 2021-03-24T20:41:38.754 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-03-24T20:41:38.756 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:38.781 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:38 +0000 (0:00:00.106) 0:02:21.026 ******* 2021-03-24T20:41:38.851 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:38.874 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:38 +0000 (0:00:00.092) 0:02:21.119 ******* 2021-03-24T20:41:38.942 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-03-24T20:41:38.944 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:38.966 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:38 +0000 (0:00:00.092) 0:02:21.212 ******* 2021-03-24T20:41:39.035 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-03-24T20:41:39.037 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.059 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:39 +0000 (0:00:00.092) 0:02:21.304 ******* 2021-03-24T20:41:39.128 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-03-24T20:41:39.130 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.152 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:39 +0000 (0:00:00.093) 0:02:21.398 ******* 2021-03-24T20:41:39.160 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-24T20:41:39.160 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HTKcMyPuR5IW5QgeLWqGifpassD1DbHf6U'], 'uuids': ['06ae5a2f-ed6f-4e04-98e7-be983a7c9162']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.171847", "end": "2021-03-24 20:41:38.731786", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HTKcMyPuR5IW5QgeLWqGifpassD1DbHf6U"], "labels": [], "masters": [], "uuids": ["06ae5a2f-ed6f-4e04-98e7-be983a7c9162"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:37.559939", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T20:41:39.162 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.164 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.227 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T20:41:39.229 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-03-24T20:41:39.251 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:39 +0000 (0:00:00.099) 0:02:21.497 ******* 2021-03-24T20:41:39.342 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-03-24T20:41:39.344 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop1) => {"ansible_loop_var": "item", "changed": false, "item": "loop1", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.353 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.364 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop6) => {"ansible_loop_var": "item", "changed": false, "item": "loop6", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.366 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop4) => {"ansible_loop_var": "item", "changed": false, "item": "loop4", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.376 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.377 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.387 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-03-24T20:41:39.398 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop3) => {"ansible_loop_var": "item", "changed": false, "item": "loop3", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.400 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop2) => {"ansible_loop_var": "item", "changed": false, "item": "loop2", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.409 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.411 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop0) => {"ansible_loop_var": "item", "changed": false, "item": "loop0", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.421 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop7) => {"ansible_loop_var": "item", "changed": false, "item": "loop7", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.421 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.432 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=loop5) => {"ansible_loop_var": "item", "changed": false, "item": "loop5", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.434 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.460 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:39 +0000 (0:00:00.209) 0:02:21.706 ******* 2021-03-24T20:41:39.708 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-03-24T20:41:39.710 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003186", "end": "2021-03-24 20:41:39.681999", "rc": 0, "start": "2021-03-24 20:41:39.678813", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/virtualenv/bin/teuthology-lock"]} 2021-03-24T20:41:39.734 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:39 +0000 (0:00:00.273) 0:02:21.979 ******* 2021-03-24T20:41:39.813 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.838 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:39 +0000 (0:00:00.104) 0:02:22.084 ******* 2021-03-24T20:41:39.917 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-03-24T20:41:39.919 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:39.940 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:39 +0000 (0:00:00.102) 0:02:22.186 ******* 2021-03-24T20:41:40.010 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** 2021-03-24T20:41:40.012 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:40.036 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:40 +0000 (0:00:00.095) 0:02:22.281 ******* 2021-03-24T20:41:40.403 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-24T20:41:40.405 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HT970pUYplD7zUNwXB2LbJ77Msx8Vw2gEv'], 'uuids': ['88e841f0-a0bb-4fde-b1e4-84a50c7b824b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.115132", "end": "2021-03-24 20:41:39.990946", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HT970pUYplD7zUNwXB2LbJ77Msx8Vw2gEv"], "labels": [], "masters": [], "uuids": ["88e841f0-a0bb-4fde-b1e4-84a50c7b824b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:38.875814", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T20:41:40.416 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:41.015 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-03-24T20:41:41.017 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1616618465, "cache_updated": false, "changed": false} 2021-03-24T20:41:41.038 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:41 +0000 (0:00:01.002) 0:02:23.284 ******* 2021-03-24T20:41:41.108 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-03-24T20:41:41.178 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Wednesday 24 March 2021 20:41:41 +0000 (0:00:00.093) 0:02:23.378 ******* 2021-03-24T20:41:41.211 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-24T20:41:41.213 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item={'block_used': 13753819, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341289, 'size_available': 927093919744, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149510, 'block_size': 4096, 'inode_available': 60905466}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341289, "block_size": 4096, "block_total": 240095108, "block_used": 13753819, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905466, "inode_total": 61054976, "inode_used": 149510, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927093919744, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}} 2021-03-24T20:41:41.236 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:41 +0000 (0:00:00.103) 0:02:23.481 ******* 2021-03-24T20:41:41.430 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-03-24T20:41:41.432 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.010960", "end": "2021-03-24 20:41:41.252604", "rc": 0, "start": "2021-03-24 20:41:41.241644", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:41.454 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:41 +0000 (0:00:00.218) 0:02:23.700 ******* 2021-03-24T20:41:41.534 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-03-24T20:41:41.536 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'block_used': 13753819, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341289, 'size_available': 927093919744, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149510, 'block_size': 4096, 'inode_available': 60905466}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341289, "block_size": 4096, "block_total": 240095108, "block_used": 13753819, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905466, "inode_total": 61054976, "inode_used": 149510, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927093919744, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:41.559 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:41 +0000 (0:00:00.104) 0:02:23.804 ******* 2021-03-24T20:41:41.628 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-03-24T20:41:41.661 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:41 +0000 (0:00:00.100) 0:02:23.905 ******* 2021-03-24T20:41:41.667 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-24T20:41:41.668 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HTfSvBdBiqOaFyP9bx3m9CY3BGpXkOWxaF'], 'uuids': ['6d55ffed-87c7-4cbc-96d4-685633cf57e6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.113308", "end": "2021-03-24 20:41:41.242303", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HTfSvBdBiqOaFyP9bx3m9CY3BGpXkOWxaF"], "labels": [], "masters": [], "uuids": ["6d55ffed-87c7-4cbc-96d4-685633cf57e6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:40.128995", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T20:41:41.710 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:41 +0000 (0:00:00.051) 0:02:23.956 ******* 2021-03-24T20:41:41.771 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:41.867 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-24T20:41:41.869 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:42.152 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HTevsRnLTJgDpfoJoI4wwlEpqcIi58LFYt'], 'uuids': ['c31e5e1d-e590-45e2-a0c6-ed280195ce53']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.157232", "end": "2021-03-24 20:41:41.744850", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HTevsRnLTJgDpfoJoI4wwlEpqcIi58LFYt"], "labels": [], "masters": [], "uuids": ["c31e5e1d-e590-45e2-a0c6-ed280195ce53"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:41.587618", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.152499 s, 756 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.152499 s, 756 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:42.154 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:42.165 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:42.450 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HTcM7900FBnGilVFWJ9pySp92ZdLH3ddjC'], 'uuids': ['dc6551df-c4bf-4f5b-bd97-06312aa04002']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.154605", "end": "2021-03-24 20:41:42.041832", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HTcM7900FBnGilVFWJ9pySp92ZdLH3ddjC"], "labels": [], "masters": [], "uuids": ["dc6551df-c4bf-4f5b-bd97-06312aa04002"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:41.887227", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.149904 s, 769 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.149904 s, 769 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:42.452 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W2X1', 'wwn-0x5000c50087b8ff21'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4W2X1-part1', 'wwn-0x5000c50087b8ff21-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c50087b8ff21', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4W2X1', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W2X1", "wwn-0x5000c50087b8ff21"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4W2X1-part1", "wwn-0x5000c50087b8ff21-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4W2X1", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50087b8ff21"}}, "skip_reason": 2021-03-24T20:41:42.454 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-03-24T20:41:42.632 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '781422768', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-nMdmyw-mycu-nbWw-1O9F-r2Ag-7c6m-iD0Ji9', 'nvme-INTEL_SSDPEDMD400G4_CVFT5344005B400BGN', 'nvme-nvme.8086-43564654353334343030354234303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPEDMD400G4', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '372.61 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.051330", "end": "2021-03-24 20:41:42.226907", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-nMdmyw-mycu-nbWw-1O9F-r2Ag-7c6m-iD0Ji9", "nvme-INTEL_SSDPEDMD400G4_CVFT5344005B400BGN", "nvme-nvme.8086-43564654353334343030354234303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:42.175577", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0465558 s, 2.5 GB/s", "std 2021-03-24T20:41:42.634 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0465558 s, 2.5 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:42.644 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:42.656 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:42.929 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HTKcMyPuR5IW5QgeLWqGifpassD1DbHf6U'], 'uuids': ['06ae5a2f-ed6f-4e04-98e7-be983a7c9162']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.155989", "end": "2021-03-24 20:41:42.521963", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HTKcMyPuR5IW5QgeLWqGifpassD1DbHf6U"], "labels": [], "masters": [], "uuids": ["06ae5a2f-ed6f-4e04-98e7-be983a7c9162"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:42.365974", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.15294 s, 754 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.15294 s, 754 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:42.941 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:42.952 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:43.237 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HT970pUYplD7zUNwXB2LbJ77Msx8Vw2gEv'], 'uuids': ['88e841f0-a0bb-4fde-b1e4-84a50c7b824b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.156729", "end": "2021-03-24 20:41:42.825623", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HT970pUYplD7zUNwXB2LbJ77Msx8Vw2gEv"], "labels": [], "masters": [], "uuids": ["88e841f0-a0bb-4fde-b1e4-84a50c7b824b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:42.668894", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151981 s, 759 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151981 s, 759 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:43.238 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:43.526 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HTfSvBdBiqOaFyP9bx3m9CY3BGpXkOWxaF'], 'uuids': ['6d55ffed-87c7-4cbc-96d4-685633cf57e6']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.155753", "end": "2021-03-24 20:41:43.119955", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-8FCgSLsxizy6mGjkmrgqhVpujCkb90HTfSvBdBiqOaFyP9bx3m9CY3BGpXkOWxaF"], "labels": [], "masters": [], "uuids": ["6d55ffed-87c7-4cbc-96d4-685633cf57e6"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:42.964202", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.151357 s, 762 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.151357 s, 762 MB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:43.571 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:43 +0000 (0:00:01.861) 0:02:25.817 ******* 2021-03-24T20:41:43.776 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-24T20:41:43.778 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctLiGv0WO19vYM2rlLd9riOKRrYExfJDSD'], 'uuids': ['4b373812-b3d6-44e0-8094-f891d8e3f7ed']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.781275", "end": "2021-03-24 20:41:43.588755", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctLiGv0WO19vYM2rlLd9riOKRrYExfJDSD"], "labels": [], "masters": [], "uuids": ["4b373812-b3d6-44e0-8094-f891d8e3f7ed"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:41.807480", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi052.front.sepia.ceph.com] => 2021-03-24T20:41:43.779 INFO:teuthology.task.ansible.out:(item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:43.789 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:43.927 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-03-24T20:41:43.928 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.156652", "end": "2021-03-24 20:41:43.468420", "rc": 0, "start": "2021-03-24 20:41:43.311768", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:43.951 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:43 +0000 (0:00:00.379) 0:02:26.197 ******* 2021-03-24T20:41:44.158 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-03-24T20:41:44.160 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.070346", "end": "2021-03-24 20:41:43.758538", "rc": 0, "start": "2021-03-24 20:41:43.688192", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:44.183 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:44 +0000 (0:00:00.232) 0:02:26.429 ******* 2021-03-24T20:41:44.253 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-03-24T20:41:44.318 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:44 +0000 (0:00:00.134) 0:02:26.564 ******* 2021-03-24T20:41:44.347 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-03-24T20:41:44.349 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:44.372 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:44 +0000 (0:00:00.054) 0:02:26.618 ******* 2021-03-24T20:41:44.399 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-03-24T20:41:44.401 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:44.423 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:44 +0000 (0:00:00.051) 0:02:26.669 ******* 2021-03-24T20:41:44.451 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-24T20:41:44.452 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:44.476 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:44 +0000 (0:00:00.053) 0:02:26.722 ******* 2021-03-24T20:41:44.504 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-24T20:41:44.506 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:44.527 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:44 +0000 (0:00:00.051) 0:02:26.773 ******* 2021-03-24T20:41:44.663 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-03-24T20:41:44.665 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T20:41:44.690 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:44 +0000 (0:00:00.162) 0:02:26.936 ******* 2021-03-24T20:41:44.729 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-24T20:41:44.730 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'block_used': 13753810, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341298, 'size_available': 927093956608, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149511, 'block_size': 4096, 'inode_available': 60905465}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341298, "block_size": 4096, "block_total": 240095108, "block_used": 13753810, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905465, "inode_total": 61054976, "inode_used": 149511, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927093956608, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:44.753 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:44 +0000 (0:00:00.062) 0:02:26.999 ******* 2021-03-24T20:41:44.780 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-03-24T20:41:44.781 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:44.803 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:44 +0000 (0:00:00.049) 0:02:27.049 ******* 2021-03-24T20:41:44.830 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T20:41:44.832 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:44.854 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:44 +0000 (0:00:00.051) 0:02:27.100 ******* 2021-03-24T20:41:44.881 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:44.905 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:44 +0000 (0:00:00.050) 0:02:27.150 ******* 2021-03-24T20:41:44.932 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-03-24T20:41:44.934 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:44.955 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:44 +0000 (0:00:00.050) 0:02:27.201 ******* 2021-03-24T20:41:44.982 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-24T20:41:44.984 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:45.005 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:45 +0000 (0:00:00.049) 0:02:27.251 ******* 2021-03-24T20:41:45.032 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-03-24T20:41:45.034 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [testnode : Zap all non-root disks] *************************************** changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctmBWnpK7GzUc4wWMhtTR3f4352tjizKZd'], 'uuids': ['3c3155a0-d255-41be-8e13-e8b911b41f4b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.116685", "end": "2021-03-24 20:41:44.856203", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctmBWnpK7GzUc4wWMhtTR3f4352tjizKZd"], "labels": [], "masters": [], "uuids": ["3c3155a0-d255-41be-8e13-e8b911b41f4b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:43.739518", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot 2021-03-24T20:41:45.036 INFO:teuthology.task.ansible.out: or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T20:41:45.057 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:45 +0000 (0:00:00.052) 0:02:27.303 ******* 2021-03-24T20:41:45.065 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HRYS', 'wwn-0x5000c500871a48d9'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HRYS-part1', 'wwn-0x5000c500871a48d9-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871a48d9', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HRYS', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HRYS", "wwn-0x5000c500871a48d9"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HRYS-part1", "wwn-0x5000c500871a48d9-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HRYS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a48d9"}}, "skip_reason": 2021-03-24T20:41:45.067 INFO:teuthology.task.ansible.out:"Conditional result was False"} 2021-03-24T20:41:45.667 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-03-24T20:41:45.669 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-24T20:41:45.692 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:45 +0000 (0:00:00.634) 0:02:27.937 ******* 2021-03-24T20:41:46.298 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-24T20:41:46.300 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T20:41:46.310 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-24T20:41:46.312 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-iuaGY7-rC6u-PpFd-5Ym3-Udn0-MlFP-MLbFTV', 'nvme-INTEL_SSDPED1D480GA_PHMB7513010N480DGN', 'nvme-nvme.8086-50484d42373531333031304e34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.134729", "end": "2021-03-24 20:41:46.128590", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-iuaGY7-rC6u-PpFd-5Ym3-Udn0-MlFP-MLbFTV", "nvme-INTEL_SSDPED1D480GA_PHMB7513010N480DGN", "nvme-nvme.8086-50484d42373531333031304e34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:44.993861", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! You may now pa 2021-03-24T20:41:46.314 INFO:teuthology.task.ansible.out:rtition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T20:41:46.323 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:46.324 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:46.574 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T20:41:46.814 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T20:41:47.193 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T20:41:47.444 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T20:41:47.481 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:47 +0000 (0:00:01.789) 0:02:29.726 ******* 2021-03-24T20:41:47.603 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-24T20:41:47.605 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctQNx38VVQAdMqKesTzJM6qMF3WWApD01A'], 'uuids': ['cf4ea281-b307-4d1b-83d8-238dd950379e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.142197", "end": "2021-03-24 20:41:47.416725", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctQNx38VVQAdMqKesTzJM6qMF3WWApD01A"], "labels": [], "masters": [], "uuids": ["cf4ea281-b307-4d1b-83d8-238dd950379e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:46.274528", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi052.front.sepia.ceph.com] => (i 2021-03-24T20:41:47.606 INFO:teuthology.task.ansible.out:tem={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:47.616 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:47.617 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-24T20:41:47.641 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:47 +0000 (0:00:00.160) 0:02:29.886 ******* 2021-03-24T20:41:47.795 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-24T20:41:47.796 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-24T20:41:47.922 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-24T20:41:48.038 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-24T20:41:48.039 INFO:teuthology.task.ansible.out: 2021-03-24T20:41:48.156 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-24T20:41:48.167 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:48.193 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:48 +0000 (0:00:00.551) 0:02:30.438 ******* 2021-03-24T20:41:48.220 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-03-24T20:41:48.221 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:48.245 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:48 +0000 (0:00:00.052) 0:02:30.491 ******* 2021-03-24T20:41:48.272 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-03-24T20:41:48.274 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:48.296 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:48 +0000 (0:00:00.050) 0:02:30.541 ******* 2021-03-24T20:41:48.449 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-03-24T20:41:48.450 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T20:41:48.473 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:48 +0000 (0:00:00.177) 0:02:30.719 ******* 2021-03-24T20:41:48.511 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-03-24T20:41:48.513 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-24T20:41:48.539 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:48 +0000 (0:00:00.066) 0:02:30.785 ******* 2021-03-24T20:41:48.997 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-24T20:41:48.999 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctXizJSvf1kdPDrfLes7fr0XziZZaphJeo'], 'uuids': ['d0612f09-5b76-434a-80b9-956eba699269']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.129270", "end": "2021-03-24 20:41:48.692960", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctXizJSvf1kdPDrfLes7fr0XziZZaphJeo"], "labels": [], "masters": [], "uuids": ["d0612f09-5b76-434a-80b9-956eba699269"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:47.563690", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} skipping: [smithi052.front.sepia.ceph.com] => 2021-03-24T20:41:49.000 INFO:teuthology.task.ansible.out:(item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:49.062 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-03-24T20:41:49.064 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true} 2021-03-24T20:41:49.087 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:49 +0000 (0:00:00.548) 0:02:31.333 ******* 2021-03-24T20:41:49.450 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-03-24T20:41:49.451 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-03-24T20:41:49.473 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:49 +0000 (0:00:00.385) 0:02:31.719 ******* 2021-03-24T20:41:49.500 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-03-24T20:41:49.502 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:49.524 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:49 +0000 (0:00:00.050) 0:02:31.770 ******* 2021-03-24T20:41:49.551 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-03-24T20:41:49.553 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:49.574 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:49 +0000 (0:00:00.050) 0:02:31.820 ******* 2021-03-24T20:41:49.601 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:49.625 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:49 +0000 (0:00:00.050) 0:02:31.870 ******* 2021-03-24T20:41:49.652 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:49.675 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:49 +0000 (0:00:00.050) 0:02:31.921 ******* 2021-03-24T20:41:49.828 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-03-24T20:41:49.830 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-03-24T20:41:49.853 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:49 +0000 (0:00:00.177) 0:02:32.098 ******* 2021-03-24T20:41:50.006 INFO:teuthology.task.ansible.out: 2021-03-24T20:41:50.007 INFO:teuthology.task.ansible.out:TASK [testnode : Add CPAN config directory for the root user.] ***************** ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T20:41:50.030 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:50 +0000 (0:00:00.176) 0:02:32.275 ******* 2021-03-24T20:41:50.148 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-03-24T20:41:50.149 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctdsRVDJy8BiMRIV0OP47hWkKDDkg16kEL'], 'uuids': ['5e513be0-c4c6-4e7b-a14c-b49f5d32a590']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.136213", "end": "2021-03-24 20:41:49.968876", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctdsRVDJy8BiMRIV0OP47hWkKDDkg16kEL"], "labels": [], "masters": [], "uuids": ["5e513be0-c4c6-4e7b-a14c-b49f5d32a590"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:48.832663", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-03-24T20:41:50.184 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:50 +0000 (0:00:00.153) 0:02:32.429 ******* 2021-03-24T20:41:50.234 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-24T20:41:50.236 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop1', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop1", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:50.382 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-03-24T20:41:50.384 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-03-24T20:41:50.407 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:50 +0000 (0:00:00.223) 0:02:32.653 ******* 2021-03-24T20:41:50.477 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-24T20:41:50.479 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctLiGv0WO19vYM2rlLd9riOKRrYExfJDSD'], 'uuids': ['4b373812-b3d6-44e0-8094-f891d8e3f7ed']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.109896", "end": "2021-03-24 20:41:50.296039", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctLiGv0WO19vYM2rlLd9riOKRrYExfJDSD"], "labels": [], "masters": [], "uuids": ["4b373812-b3d6-44e0-8094-f891d8e3f7ed"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:50.186143", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.105221 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.105221 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop6', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': 2021-03-24T20:41:50.481 INFO:teuthology.task.ansible.out: '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop6", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:50.490 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop4', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop4", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:50.733 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctmBWnpK7GzUc4wWMhtTR3f4352tjizKZd'], 'uuids': ['3c3155a0-d255-41be-8e13-e8b911b41f4b']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.111548", "end": "2021-03-24 20:41:50.548816", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctmBWnpK7GzUc4wWMhtTR3f4352tjizKZd"], "labels": [], "masters": [], "uuids": ["3c3155a0-d255-41be-8e13-e8b911b41f4b"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:50.437268", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.106737 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.106737 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:50.744 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'scheduler_mode': 'cfq', 'rotational': '1', 'vendor': 'ATA', 'sectors': '1953525168', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HRYS', 'wwn-0x5000c500871a48d9'], 'uuids': []}, 'partitions': {'sda1': {'sectorsize': 512, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'links': {'masters': [], 'labels': [], 'ids': ['ata-ST1000NM0033-9ZM173_Z1W4HRYS-part1', 'wwn-0x5000c500871a48d9-part1'], 'uuids': ['c2492a7c-537b-4f45-893c-50f733187038']}, 'sectors': '1953522688', 'start': '2048', 'holders': [], 'size': '931.51 GB'}}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'holders': [], 'wwn': '0x5000c500871a48d9', 'model': 'ST1000NM0033-9ZM', 'serial': 'Z1W4HRYS', 'size': '931.51 GB'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HRYS", "wwn-0x5000c500871a48d9"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W4HRYS-part1", "wwn-0x5000c500871a48d9-part1"], "labels": [], "masters": [], "uuids": ["c2492a7c-537b-4f45-893c-50f733187038"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "cfq", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W4HRYS", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c500871a48d9"}}, "skip_reason": 2021-03-24T20:41:50.746 INFO:teuthology.task.ansible.out:"Conditional result was False"} TASK [testnode : Upload CPAN config for root.] ********************************* ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-03-24T20:41:50.769 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:50 +0000 (0:00:00.362) 0:02:33.015 ******* 2021-03-24T20:41:50.922 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-03-24T20:41:50.924 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'scheduler_mode': 'none', 'rotational': '0', 'vendor': None, 'sectors': '937703088', 'links': {'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4'], 'labels': [], 'ids': ['lvm-pv-uuid-iuaGY7-rC6u-PpFd-5Ym3-Udn0-MlFP-MLbFTV', 'nvme-INTEL_SSDPED1D480GA_PHMB7513010N480DGN', 'nvme-nvme.8086-50484d42373531333031304e34383044474e-494e54454c2053534450454431443438304741-00000001'], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': 'Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': 'INTEL SSDPED1D480GA', 'partitions': {}, 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4'], 'size': '447.13 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.048897", "end": "2021-03-24 20:41:50.745066", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation Optane SSD 900P Series", "links": {"ids": ["lvm-pv-uuid-iuaGY7-rC6u-PpFd-5Ym3-Udn0-MlFP-MLbFTV", "nvme-INTEL_SSDPED1D480GA_PHMB7513010N480DGN", "nvme-nvme.8086-50484d42373531333031304e34383044474e-494e54454c2053534450454431443438304741-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPED1D480GA", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "937703088", "sectorsize": "512", "size": "447.13 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:50.696169", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0443298 s, 2.6 GB/s", "stderr_lines": [" 2021-03-24T20:41:50.926 INFO:teuthology.task.ansible.out:110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0443298 s, 2.6 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:50.935 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop3', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop3", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:50.947 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop2', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop2", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:51.170 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '31252480', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctQNx38VVQAdMqKesTzJM6qMF3WWApD01A'], 'uuids': ['cf4ea281-b307-4d1b-83d8-238dd950379e']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '14.90 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.106037", "end": "2021-03-24 20:41:50.992213", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctQNx38VVQAdMqKesTzJM6qMF3WWApD01A"], "labels": [], "masters": [], "uuids": ["cf4ea281-b307-4d1b-83d8-238dd950379e"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:50.886176", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.102078 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.102078 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:51.182 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop0', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop0", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:51.183 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop7', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop7", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:51.427 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctXizJSvf1kdPDrfLes7fr0XziZZaphJeo'], 'uuids': ['d0612f09-5b76-434a-80b9-956eba699269']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.108945", "end": "2021-03-24 20:41:51.246937", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctXizJSvf1kdPDrfLes7fr0XziZZaphJeo"], "labels": [], "masters": [], "uuids": ["d0612f09-5b76-434a-80b9-956eba699269"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:51.137992", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.105215 s, 1.1 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.105215 s, 1.1 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:51.428 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'loop5', 'value': {'scheduler_mode': 'none', 'rotational': '1', 'vendor': None, 'sectors': '0', 'links': {'masters': [], 'labels': [], 'ids': [], 'uuids': []}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '0', 'model': None, 'partitions': {}, 'holders': [], 'size': '0.00 Bytes'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "loop5", "value": {"holders": [], "host": "", "links": {"ids": [], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "0", "sectorsize": "512", "size": "0.00 Bytes", "support_discard": "0", "vendor": null, "virtual": 1}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:51.661 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'scheduler_mode': '', 'rotational': '0', 'vendor': None, 'sectors': '187490304', 'links': {'masters': [], 'labels': [], 'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctdsRVDJy8BiMRIV0OP47hWkKDDkg16kEL'], 'uuids': ['5e513be0-c4c6-4e7b-a14c-b49f5d32a590']}, 'sas_device_handle': None, 'sas_address': None, 'virtual': 1, 'host': '', 'sectorsize': '512', 'removable': '0', 'support_discard': '512', 'model': None, 'partitions': {}, 'holders': [], 'size': '89.40 GB'}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.099723", "end": "2021-03-24 20:41:51.480851", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-TBzgR7JXdlKdb88khki0XsaWH5ibZUctdsRVDJy8BiMRIV0OP47hWkKDDkg16kEL"], "labels": [], "masters": [], "uuids": ["5e513be0-c4c6-4e7b-a14c-b49f5d32a590"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-03-24 20:41:51.381128", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0971205 s, 1.2 GB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0971205 s, 1.2 GB/s"], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:51.700 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:51 +0000 (0:00:00.930) 0:02:33.945 ******* 2021-03-24T20:41:51.788 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** ok: [smithi063.front.sepia.ceph.com] => {"cache_update_time": 1616618456, "cache_updated": false, "changed": false} Wednesday 24 March 2021 20:41:51 +0000 (0:00:00.042) 0:02:33.988 ******* 2021-03-24T20:41:51.929 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-03-24T20:41:51.931 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.048445", "end": "2021-03-24 20:41:51.532233", "rc": 0, "start": "2021-03-24 20:41:51.483788", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-03-24T20:41:51.955 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:51 +0000 (0:00:00.212) 0:02:34.200 ******* 2021-03-24T20:41:51.983 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-03-24T20:41:51.985 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:52.007 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:52 +0000 (0:00:00.051) 0:02:34.252 ******* 2021-03-24T20:41:52.023 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-03-24T20:41:52.025 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.170388", "end": "2021-03-24 20:41:51.832466", "rc": 0, "start": "2021-03-24 20:41:51.662078", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:52.048 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:52 +0000 (0:00:00.041) 0:02:34.294 ******* 2021-03-24T20:41:52.055 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-03-24T20:41:52.058 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:52.082 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:52 +0000 (0:00:00.033) 0:02:34.328 ******* 2021-03-24T20:41:52.346 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-03-24T20:41:52.348 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.041687", "end": "2021-03-24 20:41:52.175839", "rc": 0, "start": "2021-03-24 20:41:52.134152", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:52.372 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:52 +0000 (0:00:00.290) 0:02:34.618 ******* 2021-03-24T20:41:52.400 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-03-24T20:41:52.426 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:52 +0000 (0:00:00.053) 0:02:34.672 ******* 2021-03-24T20:41:52.453 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* 2021-03-24T20:41:52.455 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:52.480 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:52 +0000 (0:00:00.053) 0:02:34.725 ******* 2021-03-24T20:41:52.507 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-03-24T20:41:52.509 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:52.530 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:52 +0000 (0:00:00.050) 0:02:34.776 ******* 2021-03-24T20:41:52.558 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:52.583 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:52 +0000 (0:00:00.052) 0:02:34.829 ******* 2021-03-24T20:41:52.611 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-03-24T20:41:52.613 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:52.640 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:52 +0000 (0:00:00.057) 0:02:34.886 ******* 2021-03-24T20:41:52.790 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-03-24T20:41:52.791 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-03-24T20:41:52.814 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:52 +0000 (0:00:00.173) 0:02:35.059 ******* 2021-03-24T20:41:52.851 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-03-24T20:41:52.853 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'block_used': 13753819, 'uuid': 'c2492a7c-537b-4f45-893c-50f733187038', 'size_total': 983429562368, 'block_total': 240095108, 'mount': '/', 'block_available': 226341289, 'size_available': 927093919744, 'fstype': 'ext4', 'inode_total': 61054976, 'options': 'rw,relatime,errors=remount-ro,data=ordered', 'device': '/dev/sda1', 'inode_used': 149510, 'block_size': 4096, 'inode_available': 60905466}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 226341289, "block_size": 4096, "block_total": 240095108, "block_used": 13753819, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60905466, "inode_total": 61054976, "inode_used": 149510, "mount": "/", "options": "rw,relatime,errors=remount-ro,data=ordered", "size_available": 927093919744, "size_total": 983429562368, "uuid": "c2492a7c-537b-4f45-893c-50f733187038"}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:52.876 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:52 +0000 (0:00:00.062) 0:02:35.121 ******* 2021-03-24T20:41:52.913 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-03-24T20:41:52.915 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:52.937 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:52 +0000 (0:00:00.061) 0:02:35.183 ******* 2021-03-24T20:41:52.975 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T20:41:52.976 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:52.998 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:52 +0000 (0:00:00.061) 0:02:35.244 ******* 2021-03-24T20:41:53.025 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:53.050 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:53 +0000 (0:00:00.051) 0:02:35.295 ******* 2021-03-24T20:41:53.057 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-03-24T20:41:53.059 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"cache_update_time": 1616618456, "cache_updated": false, "changed": false} 2021-03-24T20:41:53.084 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:53 +0000 (0:00:00.033) 0:02:35.329 ******* 2021-03-24T20:41:53.090 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-03-24T20:41:53.092 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:53.117 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:53 +0000 (0:00:00.033) 0:02:35.362 ******* 2021-03-24T20:41:53.124 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-03-24T20:41:53.126 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:53.151 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:53 +0000 (0:00:00.034) 0:02:35.397 ******* 2021-03-24T20:41:53.157 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* 2021-03-24T20:41:53.160 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:53.186 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:53 +0000 (0:00:00.034) 0:02:35.432 ******* 2021-03-24T20:41:53.214 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-03-24T20:41:53.216 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:53.239 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:53 +0000 (0:00:00.052) 0:02:35.485 ******* 2021-03-24T20:41:53.493 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-03-24T20:41:53.526 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:53 +0000 (0:00:00.287) 0:02:35.772 ******* 2021-03-24T20:41:53.557 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-03-24T20:41:53.559 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:53.582 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:53 +0000 (0:00:00.056) 0:02:35.828 ******* 2021-03-24T20:41:53.591 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-03-24T20:41:53.594 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-03-24T20:41:53.627 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:53 +0000 (0:00:00.044) 0:02:35.873 ******* 2021-03-24T20:41:53.810 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-03-24T20:41:53.812 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-03-24 20:39:13 UTC", "ActiveEnterTimestampMonotonic": "24752158", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "sysinit.target system.slice systemd-tmpfiles-setup.service -.mount basic.target systemd-journald.socket network.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-03-24 20:39:06 UTC", "AssertTimestampMonotonic": "18200368", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-03-24 20:39:06 UTC", "ConditionTimestampMonotonic": "18200367", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-03-24T20:41:53.814 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1988", "ExecMainStartTimestamp": "Wed 2021-03-24 20:39:13 UTC", "ExecMainStartTimestampMonotonic": "24752132", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-03-24 20:39:06 UTC", "InactiveExitTimestampMonotonic": "18201067", "InvocationID": "5ac8ef447c4c49ef9a2c078e6cc5d4fb", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128029", "LimitNPROCSoft": "128029", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-03-24T20:41:53.816 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "128029", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1988", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp /tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-03-24 20:39:13 UTC", "S 2021-03-24T20:41:53.817 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "24752158", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-03-24 20:39:13 UTC", "WatchdogTimestampMonotonic": "24752156", "WatchdogUSec": "0"}} 2021-03-24T20:41:53.842 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:53 +0000 (0:00:00.214) 0:02:36.087 ******* 2021-03-24T20:41:53.872 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T20:41:53.874 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-24T20:41:53.906 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:53 +0000 (0:00:00.064) 0:02:36.152 ******* 2021-03-24T20:41:53.938 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T20:41:53.939 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:53.967 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:53 +0000 (0:00:00.060) 0:02:36.212 ******* 2021-03-24T20:41:53.975 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-24T20:41:53.977 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T20:41:53.998 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-03-24T20:41:54.000 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:54.023 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:54 +0000 (0:00:00.056) 0:02:36.269 ******* 2021-03-24T20:41:54.180 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-03-24T20:41:54.182 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T20:41:54.415 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T20:41:54.668 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T20:41:54.926 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-03-24T20:41:54.957 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:54 +0000 (0:00:00.933) 0:02:37.202 ******* 2021-03-24T20:41:54.963 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-03-24T20:41:54.965 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"cache_update_time": 1616618456, "cache_updated": false, "changed": false} 2021-03-24T20:41:55.000 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:54 +0000 (0:00:00.043) 0:02:37.246 ******* 2021-03-24T20:41:55.029 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-03-24T20:41:55.031 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:55.056 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:55 +0000 (0:00:00.055) 0:02:37.301 ******* 2021-03-24T20:41:55.106 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-03-24T20:41:55.107 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-03-24T20:41:55.132 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:55 +0000 (0:00:00.076) 0:02:37.377 ******* 2021-03-24T20:41:55.202 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-03-24T20:41:55.204 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-03-24T20:41:55.242 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:55 +0000 (0:00:00.110) 0:02:37.488 ******* 2021-03-24T20:41:55.317 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-24T20:41:55.318 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-24T20:41:55.457 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-24T20:41:55.570 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-24T20:41:55.624 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-03-24T20:41:55.626 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-03-24T20:41:55.649 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:55 +0000 (0:00:00.406) 0:02:37.895 ******* 2021-03-24T20:41:55.691 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-03-24T20:41:55.693 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-03-24T20:41:55.704 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:55.736 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:55 +0000 (0:00:00.086) 0:02:37.982 ******* 2021-03-24T20:41:55.764 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-03-24T20:41:55.767 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:55.790 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:55 +0000 (0:00:00.053) 0:02:38.035 ******* 2021-03-24T20:41:55.818 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-03-24T20:41:55.818 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:55.842 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:55 +0000 (0:00:00.052) 0:02:38.088 ******* 2021-03-24T20:41:55.985 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-03-24T20:41:55.987 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T20:41:56.009 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:56 +0000 (0:00:00.166) 0:02:38.254 ******* 2021-03-24T20:41:56.047 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-03-24T20:41:56.049 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-03-24T20:41:56.071 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:56 +0000 (0:00:00.062) 0:02:38.317 ******* 2021-03-24T20:41:56.237 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-03-24T20:41:56.239 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true} 2021-03-24T20:41:56.269 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:56 +0000 (0:00:00.197) 0:02:38.514 ******* 2021-03-24T20:41:56.483 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-03-24T20:41:56.484 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-03-24T20:41:56.510 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:56 +0000 (0:00:00.241) 0:02:38.756 ******* 2021-03-24T20:41:56.538 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-03-24T20:41:56.540 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:56.563 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:56 +0000 (0:00:00.052) 0:02:38.809 ******* 2021-03-24T20:41:56.601 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** 2021-03-24T20:41:56.603 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:56.626 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:56 +0000 (0:00:00.062) 0:02:38.871 ******* 2021-03-24T20:41:56.634 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-03-24T20:41:56.636 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-03-24T20:41:56.660 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:56 +0000 (0:00:00.034) 0:02:38.906 ******* 2021-03-24T20:41:56.667 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** 2021-03-24T20:41:56.670 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:56.697 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:56 +0000 (0:00:00.037) 0:02:38.943 ******* 2021-03-24T20:41:56.725 INFO:teuthology.task.ansible.out: TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:56.752 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:56 +0000 (0:00:00.055) 0:02:38.998 ******* 2021-03-24T20:41:56.920 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the teuthology user.] *********** 2021-03-24T20:41:56.922 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 1000} 2021-03-24T20:41:56.946 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:56 +0000 (0:00:00.193) 0:02:39.191 ******* 2021-03-24T20:41:57.089 INFO:teuthology.task.ansible.out: TASK [testnode : Add CPAN config directory for the root user.] ***************** 2021-03-24T20:41:57.090 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T20:41:57.113 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:57 +0000 (0:00:00.167) 0:02:39.358 ******* 2021-03-24T20:41:57.235 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-03-24T20:41:57.237 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false} 2021-03-24T20:41:57.261 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:57 +0000 (0:00:00.148) 0:02:39.507 ******* 2021-03-24T20:41:57.289 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-03-24T20:41:57.291 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-03-24T20:41:57.313 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:57 +0000 (0:00:00.051) 0:02:39.559 ******* 2021-03-24T20:41:57.459 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** 2021-03-24T20:41:57.461 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "gid": 1238, "group": "ubuntu", "mode": "0755", "owner": "ubuntu", "path": "/home/ubuntu/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 1000} 2021-03-24T20:41:57.461 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-03-24T20:41:57.461 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1616618128.6714108, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1616618128.6674109, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 262873, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1616618128.6674109, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 71, "uid": 0, "version": "3476867264", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-03-24T20:41:57.490 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:57 +0000 (0:00:00.176) 0:02:39.736 ******* 2021-03-24T20:41:57.516 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:57 +0000 (0:00:00.025) 0:02:39.762 ******* 2021-03-24T20:41:57.787 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-03-24T20:41:57.789 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-03-24T20:41:57.813 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:57 +0000 (0:00:00.296) 0:02:40.058 ******* 2021-03-24T20:41:57.840 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* 2021-03-24T20:41:57.841 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "dcd55d202d82b6e06d0555630d7a8b92d7100c36", "dest": "/root/.cpan/CPAN/MyConfig.pm", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/root/.cpan/CPAN/MyConfig.pm", "size": 2117, "state": "file", "uid": 0} 2021-03-24T20:41:57.864 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:57 +0000 (0:00:00.051) 0:02:40.110 ******* 2021-03-24T20:41:57.872 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-03-24T20:41:57.873 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-24T20:41:57.900 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:57 +0000 (0:00:00.036) 0:02:40.146 ******* 2021-03-24T20:41:57.928 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-03-24T20:41:57.930 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:57.952 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:57 +0000 (0:00:00.051) 0:02:40.198 ******* 2021-03-24T20:41:58.104 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-03-24T20:41:58.106 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-24T20:41:58.129 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:58 +0000 (0:00:00.176) 0:02:40.374 ******* 2021-03-24T20:41:58.261 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-03-24T20:41:58.263 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-24T20:41:58.286 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:58 +0000 (0:00:00.156) 0:02:40.531 ******* 2021-03-24T20:41:58.312 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-03-24T20:41:58.314 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:58.337 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:58 +0000 (0:00:00.051) 0:02:40.583 ******* 2021-03-24T20:41:58.387 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-03-24T20:41:58.389 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:58.399 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:58.401 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi063.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:58.410 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-03-24T20:41:58.437 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:58 +0000 (0:00:00.099) 0:02:40.682 ******* 2021-03-24T20:41:58.485 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-03-24T20:41:58.487 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi063.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:58.497 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:58.508 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:58.510 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:58.545 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:58 +0000 (0:00:00.108) 0:02:40.791 ******* 2021-03-24T20:41:58.572 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:58.603 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:58 +0000 (0:00:00.057) 0:02:40.849 ******* 2021-03-24T20:41:58.630 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-03-24T20:41:58.633 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:58.654 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:58 +0000 (0:00:00.050) 0:02:40.899 ******* 2021-03-24T20:41:58.680 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-03-24T20:41:58.682 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:58.708 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:58 +0000 (0:00:00.054) 0:02:40.954 ******* 2021-03-24T20:41:58.735 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-03-24T20:41:58.737 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:58.761 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:58 +0000 (0:00:00.052) 0:02:41.006 ******* 2021-03-24T20:41:58.788 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-03-24T20:41:58.790 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:58.815 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:58 +0000 (0:00:00.054) 0:02:41.061 ******* 2021-03-24T20:41:58.822 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** 2021-03-24T20:41:58.824 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1616618465, "cache_updated": false, "changed": false} 2021-03-24T20:41:58.848 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:58 +0000 (0:00:00.033) 0:02:41.094 ******* 2021-03-24T20:41:58.856 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-24T20:41:58.858 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:58.884 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:58 +0000 (0:00:00.035) 0:02:41.130 ******* 2021-03-24T20:41:58.912 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-03-24T20:41:58.914 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:58.943 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:58 +0000 (0:00:00.058) 0:02:41.188 ******* 2021-03-24T20:41:58.973 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-24T20:41:58.975 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:59.003 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:59 +0000 (0:00:00.060) 0:02:41.249 ******* 2021-03-24T20:41:59.035 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-03-24T20:41:59.037 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": ["perldoc", "-l", "Amazon::S3"], "delta": "0:00:00.046973", "end": "2021-03-24 20:41:58.868138", "rc": 0, "start": "2021-03-24 20:41:58.821165", "stderr": "", "stderr_lines": [], "stdout": "/usr/local/share/perl/5.26.1/Amazon/S3.pm", "stdout_lines": ["/usr/local/share/perl/5.26.1/Amazon/S3.pm"]} 2021-03-24T20:41:59.062 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:59 +0000 (0:00:00.058) 0:02:41.307 ******* 2021-03-24T20:41:59.096 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:59.126 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:59 +0000 (0:00:00.064) 0:02:41.372 ******* 2021-03-24T20:41:59.154 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:41:59.180 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:59 +0000 (0:00:00.053) 0:02:41.425 ******* 2021-03-24T20:41:59.189 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-03-24T20:41:59.191 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.009337", "end": "2021-03-24 20:41:58.768116", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-24 20:41:58.758779", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:59.297 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.005938", "end": "2021-03-24 20:41:58.893230", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-24 20:41:58.887292", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:59.425 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005614", "end": "2021-03-24 20:41:59.027051", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-24 20:41:59.021437", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:59.427 INFO:teuthology.task.ansible.out: 2021-03-24T20:41:59.552 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.007335", "end": "2021-03-24 20:41:59.147580", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-24 20:41:59.140245", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:41:59.577 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:59 +0000 (0:00:00.397) 0:02:41.823 ******* 2021-03-24T20:41:59.610 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-24T20:41:59.633 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:41:59 +0000 (0:00:00.056) 0:02:41.879 ******* 2021-03-24T20:41:59.785 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-03-24T20:41:59.787 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-03-24T20:42:00.090 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-03-24T20:42:00.092 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1616618465, "cache_updated": false, "changed": false} 2021-03-24T20:42:00.113 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:00 +0000 (0:00:00.479) 0:02:42.358 ******* 2021-03-24T20:42:00.140 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-03-24T20:42:00.141 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:00.164 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:00 +0000 (0:00:00.050) 0:02:42.409 ******* 2021-03-24T20:42:00.507 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ 2021-03-24T20:42:00.509 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "645fd5116197cd9da73b328bc11bf633a40d80e1", "dest": "/etc/ntp.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/ntp.conf", "size": 2767, "state": "file", "uid": 0} 2021-03-24T20:42:00.538 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:00 +0000 (0:00:00.374) 0:02:42.784 ******* 2021-03-24T20:42:00.566 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-03-24T20:42:00.568 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:00.593 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:00 +0000 (0:00:00.054) 0:02:42.838 ******* 2021-03-24T20:42:00.810 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-03-24T20:42:00.812 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "ntp", "state": "started", "status": {"ActiveEnterTimestamp": "Wed 2021-03-24 20:38:50 UTC", "ActiveEnterTimestampMonotonic": "23652466", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "basic.target network.target systemd-tmpfiles-setup.service sysinit.target -.mount systemd-journald.socket system.slice", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Wed 2021-03-24 20:38:43 UTC", "AssertTimestampMonotonic": "17149702", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Wed 2021-03-24 20:38:43 UTC", "ConditionTimestampMonotonic": "17149702", "ConfigurationDirectoryMode": "0755", "Conflicts": "systemd-timesyncd.service shutdown.target", "ControlGroup": "/system.slice/ntp.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Network Time Service", "DevicePolicy": "auto", "Documen 2021-03-24T20:42:00.814 INFO:teuthology.task.ansible.out:tation": "man:ntpd(8)", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "1862", "ExecMainStartTimestamp": "Wed 2021-03-24 20:38:50 UTC", "ExecMainStartTimestampMonotonic": "23652453", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/lib/ntp/ntp-systemd-wrapper ; argv[]=/usr/lib/ntp/ntp-systemd-wrapper ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/ntp.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "ntp.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Wed 2021-03-24 20:38:43 UTC", "InactiveExitTimestampMonotonic": "17150880", "InvocationID": "16dd8805e12344d9ad57ee5202f5dfce", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "0", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "4096", "LimitNOFILESoft": "1024", "LimitNPROC": "128053", "LimitNPROCSoft": "128053", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "Limi 2021-03-24T20:42:00.816 INFO:teuthology.task.ansible.out:tRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "1862", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "ntp.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/tmp /var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-03-24 20:38:50 UTC", "S 2021-03-24T20:42:00.818 INFO:teuthology.task.ansible.out:tateChangeTimestampMonotonic": "23652466", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "2", "TasksMax": "4915", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Wed 2021-03-24 20:38:50 UTC", "WatchdogTimestampMonotonic": "23652464", "WatchdogUSec": "0"}} 2021-03-24T20:42:00.836 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:00 +0000 (0:00:00.242) 0:02:43.081 ******* 2021-03-24T20:42:00.864 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T20:42:00.865 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-03-24T20:42:00.898 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:00 +0000 (0:00:00.062) 0:02:43.144 ******* 2021-03-24T20:42:00.926 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T20:42:00.928 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:00.951 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:00 +0000 (0:00:00.053) 0:02:43.197 ******* 2021-03-24T20:42:00.979 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* 2021-03-24T20:42:00.981 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:01.003 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:01 +0000 (0:00:00.051) 0:02:43.249 ******* 2021-03-24T20:42:01.893 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* 2021-03-24T20:42:01.895 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1616618465, "cache_updated": false, "changed": false} 2021-03-24T20:42:01.918 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:01 +0000 (0:00:00.915) 0:02:44.164 ******* 2021-03-24T20:42:01.946 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:01.970 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:01 +0000 (0:00:00.051) 0:02:44.216 ******* 2021-03-24T20:42:02.115 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-03-24T20:42:02.117 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "size": 4096, "state": "directory", "uid": 1000} 2021-03-24T20:42:02.139 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:02 +0000 (0:00:00.169) 0:02:44.385 ******* 2021-03-24T20:42:02.603 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-03-24T20:42:02.605 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1238, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "size": 73, "state": "file", "uid": 1000} 2021-03-24T20:42:02.627 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:02 +0000 (0:00:00.487) 0:02:44.872 ******* 2021-03-24T20:42:03.466 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-03-24T20:42:03.468 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "cmd": ["/usr/bin/pip", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "", "stderr_lines": [], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages\nRequirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python2.7/dist-packages", "Requirement already satisfied: execnet in /usr/local/lib/python2.7/dist-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python2.7/dist-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-03-24T20:42:03.493 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:03 +0000 (0:00:00.866) 0:02:45.739 ******* 2021-03-24T20:42:04.025 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* ok: [smithi052.front.sepia.ceph.com] => {"changed": false} 2021-03-24T20:42:04.053 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:04 +0000 (0:00:00.559) 0:02:46.298 ******* 2021-03-24T20:42:04.101 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-03-24T20:42:04.103 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"interface": "ens1f0"}, "changed": false} 2021-03-24T20:42:04.127 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:04 +0000 (0:00:00.073) 0:02:46.372 ******* 2021-03-24T20:42:04.280 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-03-24T20:42:04.282 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "stat": {"atime": 1616618092.7344677, "attr_flags": "e", "attributes": ["extents"], "block_size": 4096, "blocks": 8, "charset": "us-ascii", "ctime": 1616618091.2911897, "dev": 2049, "device_type": 0, "executable": false, "exists": true, "gid": 0, "gr_name": "root", "inode": 262873, "isblk": false, "ischr": false, "isdir": false, "isfifo": false, "isgid": false, "islnk": false, "isreg": true, "issock": false, "isuid": false, "mimetype": "text/plain", "mode": "0644", "mtime": 1616618091.2911897, "nlink": 1, "path": "/etc/network/interfaces", "pw_name": "root", "readable": true, "rgrp": true, "roth": true, "rusr": true, "size": 67, "uid": 0, "version": "3476867264", "wgrp": false, "woth": false, "writeable": true, "wusr": true, "xgrp": false, "xoth": false, "xusr": false}} 2021-03-24T20:42:04.304 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:04 +0000 (0:00:00.177) 0:02:46.550 ******* 2021-03-24T20:42:04.446 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-03-24T20:42:04.448 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "msg": ""} 2021-03-24T20:42:04.472 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:04 +0000 (0:00:00.167) 0:02:46.717 ******* 2021-03-24T20:42:04.499 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-03-24T20:42:04.501 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-03-24T20:42:04.523 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:04 +0000 (0:00:00.051) 0:02:46.769 ******* 2021-03-24T20:42:04.550 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-03-24T20:42:04.552 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:04.574 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:04 +0000 (0:00:00.050) 0:02:46.820 ******* 2021-03-24T20:42:04.705 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-03-24T20:42:04.707 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-03-24T20:42:04.729 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:04 +0000 (0:00:00.154) 0:02:46.975 ******* 2021-03-24T20:42:04.862 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-03-24T20:42:04.864 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-03-24T20:42:04.886 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:04 +0000 (0:00:00.156) 0:02:47.131 ******* 2021-03-24T20:42:04.913 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-03-24T20:42:04.915 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:04.945 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:04 +0000 (0:00:00.059) 0:02:47.191 ******* 2021-03-24T20:42:04.974 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-03-24T20:42:04.976 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=ens1f1) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False"} 2021-03-24T20:42:04.985 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=ens1f0) => {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False"} 2021-03-24T20:42:04.987 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} skipping: [smithi052.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} 2021-03-24T20:42:04.997 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-03-24T20:42:05.021 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:05 +0000 (0:00:00.075) 0:02:47.266 ******* 2021-03-24T20:42:05.058 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-03-24T20:42:05.060 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:05.069 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'ens1f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "ens1f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:05.071 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:05.081 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:05.082 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:05.123 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:05 +0000 (0:00:00.101) 0:02:47.369 ******* 2021-03-24T20:42:05.154 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-03-24T20:42:05.156 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:05.194 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:05 +0000 (0:00:00.070) 0:02:47.439 ******* 2021-03-24T20:42:05.225 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-03-24T20:42:05.227 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:05.250 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:05 +0000 (0:00:00.055) 0:02:47.495 ******* 2021-03-24T20:42:05.276 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-03-24T20:42:05.278 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:05.302 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:05 +0000 (0:00:00.052) 0:02:47.548 ******* 2021-03-24T20:42:05.329 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-03-24T20:42:05.331 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:05.353 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:05 +0000 (0:00:00.050) 0:02:47.598 ******* 2021-03-24T20:42:05.380 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-03-24T20:42:05.382 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:05.405 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:05 +0000 (0:00:00.052) 0:02:47.651 ******* 2021-03-24T20:42:05.433 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:05.458 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:05 +0000 (0:00:00.053) 0:02:47.704 ******* 2021-03-24T20:42:05.486 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-03-24T20:42:05.487 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:05.511 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:05 +0000 (0:00:00.052) 0:02:47.757 ******* 2021-03-24T20:42:05.538 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-03-24T20:42:05.541 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:05.562 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:05 +0000 (0:00:00.050) 0:02:47.808 ******* 2021-03-24T20:42:05.715 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-03-24T20:42:05.717 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.009170", "end": "2021-03-24 20:42:05.691245", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-03-24 20:42:05.682075", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:42:05.841 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.006015", "end": "2021-03-24 20:42:05.816451", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-03-24 20:42:05.810436", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:42:05.956 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.005591", "end": "2021-03-24 20:42:05.936988", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-03-24 20:42:05.931397", "stderr": "find: '/etc/ceph': No such file or directory", "stderr_lines": ["find: '/etc/ceph': No such file or directory"], "stdout": "", "stdout_lines": []} 2021-03-24T20:42:06.082 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.015283", "end": "2021-03-24 20:42:06.065138", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-03-24 20:42:06.049855", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:42:06.110 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:06 +0000 (0:00:00.547) 0:02:48.356 ******* 2021-03-24T20:42:06.140 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-03-24T20:42:06.167 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:06 +0000 (0:00:00.056) 0:02:48.413 ******* 2021-03-24T20:42:06.309 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-03-24T20:42:06.311 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "size": 0, "state": "file", "uid": 0} 2021-03-24T20:42:06.379 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-03-24T20:42:06.397 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:06 +0000 (0:00:00.229) 0:02:48.643 ******* 2021-03-24T20:42:06.419 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:06 +0000 (0:00:00.021) 0:02:48.664 ******* 2021-03-24T20:42:07.148 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi063.front.sepia.ceph.com] 2021-03-24T20:42:07.181 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] 2021-03-24T20:42:07.220 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:07 +0000 (0:00:00.801) 0:02:49.466 ******* 2021-03-24T20:42:07.253 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:07 +0000 (0:00:00.032) 0:02:49.498 ******* 2021-03-24T20:42:07.259 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-24T20:42:07.261 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-03-24T20:42:07.285 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:07 +0000 (0:00:00.032) 0:02:49.531 ******* 2021-03-24T20:42:07.292 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "apt"}, "changed": false} 2021-03-24T20:42:07.318 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:07 +0000 (0:00:00.032) 0:02:49.563 ******* 2021-03-24T20:42:07.324 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-24T20:42:07.325 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:07.349 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:07 +0000 (0:00:00.030) 0:02:49.594 ******* 2021-03-24T20:42:07.356 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:07.381 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:07 +0000 (0:00:00.032) 0:02:49.627 ******* 2021-03-24T20:42:07.388 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-03-24T20:42:07.390 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-03-24T20:42:07.415 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:07 +0000 (0:00:00.033) 0:02:49.661 ******* 2021-03-24T20:42:07.421 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml) => {"ansible_facts": {"container_packages": ["docker.io"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/apt_systems.yml"} 2021-03-24T20:42:07.449 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:07 +0000 (0:00:00.033) 0:02:49.694 ******* 2021-03-24T20:42:08.354 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-03-24T20:42:08.356 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"cache_update_time": 1616618456, "cache_updated": false, "changed": false} 2021-03-24T20:42:08.389 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:08 +0000 (0:00:00.940) 0:02:50.635 ******* 2021-03-24T20:42:08.408 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"cache_update_time": 1616618465, "cache_updated": false, "changed": false} 2021-03-24T20:42:08.440 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:08 +0000 (0:00:00.051) 0:02:50.686 ******* 2021-03-24T20:42:08.451 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-24T20:42:08.453 INFO:teuthology.task.ansible.out:skipping: [smithi063.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:08.477 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:08 +0000 (0:00:00.036) 0:02:50.722 ******* 2021-03-24T20:42:08.484 INFO:teuthology.task.ansible.out:skipping: [smithi052.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-03-24T20:42:08.510 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:08 +0000 (0:00:00.032) 0:02:50.755 ******* 2021-03-24T20:42:08.517 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-03-24T20:42:08.519 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-03-24T20:42:08.544 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:08 +0000 (0:00:00.034) 0:02:50.789 ******* 2021-03-24T20:42:08.551 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/docker/daemon.json"}, "changed": false} 2021-03-24T20:42:08.578 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:08 +0000 (0:00:00.033) 0:02:50.823 ******* 2021-03-24T20:42:08.711 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-03-24T20:42:08.713 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T20:42:08.755 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T20:42:08.833 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T20:42:08.861 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:08 +0000 (0:00:00.283) 0:02:51.107 ******* 2021-03-24T20:42:08.880 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "size": 4096, "state": "directory", "uid": 0} 2021-03-24T20:42:08.917 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:08 +0000 (0:00:00.056) 0:02:51.163 ******* 2021-03-24T20:42:09.264 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-03-24T20:42:09.266 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616618528.9261925-11260-218497782482871/source", "state": "file", "uid": 0} 2021-03-24T20:42:09.298 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616618528.9849393-11266-261295643374323/source", "state": "file", "uid": 0} 2021-03-24T20:42:09.604 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616618529.2906249-11260-193229149449229/source", "state": "file", "uid": 0} 2021-03-24T20:42:09.648 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:09 +0000 (0:00:00.730) 0:02:51.894 ******* 2021-03-24T20:42:09.657 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616618529.3269272-11266-31850080448851/source", "state": "file", "uid": 0} 2021-03-24T20:42:09.686 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:09 +0000 (0:00:00.037) 0:02:51.931 ******* 2021-03-24T20:42:13.313 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-03-24T20:42:13.315 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-43DCjp-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-43DCjp-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-03-24T20:42:13.317 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-03-24T20:42:13.339 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:13 +0000 (0:00:03.652) 0:02:55.584 ******* 2021-03-24T20:42:13.399 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "cmd": ["/usr/bin/pip2", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "", "stderr_lines": [], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-NFidZF-build\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting typing (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl\nBuilding wheels for collected packages: docopt\n Running setup.py bdist_wheel for docopt: started\n Running setup.py bdist_wheel for docopt: finished with status 'done'\n Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e\nSuccessfully built docopt\nInstalling collected packages: docopt, toml, typing, registries-conf-ctl\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-NFidZF-build", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packag 2021-03-24T20:42:13.401 INFO:teuthology.task.ansible.out:es/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting typing (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/3b/c0/e44213fcb799eac02881e2485724ba5b0914600bc9df6ed922e364fdc059/typing-3.7.4.3-py2-none-any.whl", "Building wheels for collected packages: docopt", " Running setup.py bdist_wheel for docopt: started", " Running setup.py bdist_wheel for docopt: finished with status 'done'", " Stored in directory: /root/.cache/pip/wheels/9b/04/dd/7daf4150b6d9b12949298737de9431a324d4b797ffd63f526e", "Successfully built docopt", "Installing collected packages: docopt, toml, typing, registries-conf-ctl", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2 typing-3.7.4.3"], "version": null, "virtualenv": null} 2021-03-24T20:42:13.425 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:13 +0000 (0:00:00.085) 0:02:55.670 ******* 2021-03-24T20:42:13.485 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-03-24T20:42:13.487 INFO:teuthology.task.ansible.out:ok: [smithi052.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-24T20:42:13.515 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:13 +0000 (0:00:00.090) 0:02:55.761 ******* 2021-03-24T20:42:13.575 INFO:teuthology.task.ansible.out:ok: [smithi063.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-03-24T20:42:13.601 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:13 +0000 (0:00:00.085) 0:02:55.847 ******* 2021-03-24T20:42:13.871 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/docker/daemon.json if necessary] ************ 2021-03-24T20:42:13.872 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616618533.569942-11897-74359205770829/source", "state": "file", "uid": 0} 2021-03-24T20:42:13.899 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:13 +0000 (0:00:00.297) 0:02:56.144 ******* 2021-03-24T20:42:13.970 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "checksum": "bf21a9e8fbc5a3846fb05b4fa0859e0917b2202f", "dest": "/etc/docker/daemon.json", "gid": 0, "group": "root", "md5sum": "99914b932bd37a50b983c5e7c90ae93b", "mode": "0644", "owner": "root", "size": 2, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1616618533.6570783-11917-5285227462753/source", "state": "file", "uid": 0} 2021-03-24T20:42:13.997 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:13 +0000 (0:00:00.098) 0:02:56.242 ******* 2021-03-24T20:42:14.161 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-03-24T20:42:14.163 INFO:teuthology.task.ansible.out:changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.115669", "end": "2021-03-24 20:42:14.144237", "rc": 0, "start": "2021-03-24 20:42:14.028568", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:42:14.186 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:14 +0000 (0:00:00.189) 0:02:56.432 ******* 2021-03-24T20:42:14.268 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.120847", "end": "2021-03-24 20:42:13.876898", "rc": 0, "start": "2021-03-24 20:42:13.756051", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-03-24T20:42:14.293 INFO:teuthology.task.ansible.out:Wednesday 24 March 2021 20:42:14 +0000 (0:00:00.106) 0:02:56.539 ******* 2021-03-24T20:42:17.661 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-03-24T20:42:17.663 INFO:teuthology.task.ansible.out: changed: [smithi052.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "network-online.target system.slice systemd-journald.socket basic.target docker.socket sysinit.target containerd.service firewalld.service", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "Exec 2021-03-24T20:42:17.665 INFO:teuthology.task.ansible.out:MainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128053", "LimitSIGPENDINGSoft": "128 2021-03-24T20:42:17.666 INFO:teuthology.task.ansible.out:053", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice docker.socket sysinit.target", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-03-24 20:41:10 UTC", "StateChangeTimestampMonotonic": "163470349", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", 2021-03-24T20:42:17.668 INFO:teuthology.task.ansible.out:"SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-24T20:42:17.854 INFO:teuthology.task.ansible.out:changed: [smithi063.front.sepia.ceph.com] => {"changed": true, "name": "docker", "state": "started", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "systemd-journald.socket sysinit.target basic.target system.slice firewalld.service containerd.service docker.socket network-online.target", "AllowIsolate": "no", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target", "ConsistsOf": "docker.socket", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "yes", "DelegateControllers": "cpu cpuacct io blkio memory devices pids", "Description": "Docker Application Container Engine", "DevicePolicy": "auto", "Documentation": "https://docs.docker.com", "DynamicUser": "no", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMa 2021-03-24T20:42:17.856 INFO:teuthology.task.ansible.out:inStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -s HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/lib/systemd/system/docker.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "docker.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "process", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "67108864", "LimitMEMLOCKSoft": "67108864", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "infinity", "LimitNOFILESoft": "infinity", "LimitNPROC": "infinity", "LimitNPROCSoft": "infinity", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "128029", "LimitSIGPENDINGSoft": "12802 2021-03-24T20:42:17.859 INFO:teuthology.task.ansible.out:9", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "no", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "Names": "docker.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "main", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice docker.socket sysinit.target", "Restart": "always", "RestartUSec": "2s", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "3", "StartLimitIntervalUSec": "1min", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Wed 2021-03-24 20:41:01 UTC", "StateChangeTimestampMonotonic": "132823541", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "S 2021-03-24T20:42:17.860 INFO:teuthology.task.ansible.out:uccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "infinity", "TimeoutStartUSec": "infinity", "TimeoutStopUSec": "infinity", "TimerSlackNSec": "50000", "Transient": "no", "TriggeredBy": "docker.socket", "Type": "notify", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "containerd.service network-online.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-03-24T20:42:17.936 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-03-24T20:42:17.937 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-03-24T20:42:17.939 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi052.front.sepia.ceph.com : ok=127 changed=37 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 smithi063.front.sepia.ceph.com : ok=127 changed=37 unreachable=0 failed=0 skipped=184 rescued=0 ignored=0 Wednesday 24 March 2021 20:42:17 +0000 (0:00:03.645) 0:03:00.184 ******* =============================================================================== 2021-03-24T20:42:17.940 INFO:teuthology.task.ansible.out:Create all admin users with sudo access. ------------------------------- 21.75s users : Update authorized_keys using the keys repo ---------------------- 8.47s testnode : Update apt cache. -------------------------------------------- 5.69s container-host : Install registries-conf-ctl ---------------------------- 3.65s container-host : Restart docker service --------------------------------- 3.65s testnode : Ensure packages are not present. ----------------------------- 2.20s testnode : Install packages --------------------------------------------- 1.85s Gathering Facts --------------------------------------------------------- 1.80s users : Clone the keys repo --------------------------------------------- 1.42s 2021-03-24T20:42:17.940 INFO:teuthology.task.ansible.out:testnode : Upgrade packages --------------------------------------------- 1.01s testnode : Make sure apt dependencies are installed --------------------- 1.00s testnode : Install python-apt ------------------------------------------- 0.99s container-host : Install container packages ----------------------------- 0.94s ansible-managed : Create the cephlab_sudo sudoers.d file. --------------- 0.93s testnode : Install python-pip on apt based systems. --------------------- 0.92s testnode : Install packages via pip ------------------------------------- 0.87s Gathering Facts --------------------------------------------------------- 0.82s Gathering Facts 2021-03-24T20:42:17.942 INFO:teuthology.task.ansible.out: --------------------------------------------------------- 0.80s ansible-managed : Add authorized keys for the ansible user. ------------- 0.77s testnode : Install apt keys --------------------------------------------- 0.77s 2021-03-24T20:42:18.328 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi052.front.sepia.ceph.com'), Remote(name='ubuntu@smithi063.front.sepia.ceph.com')] 2021-03-24T20:42:18.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-24T20:42:18.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi052.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T20:42:18.403 DEBUG:teuthology.orchestra.run.smithi052:> true 2021-03-24T20:42:18.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-03-24T20:42:18.609 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi063.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-03-24T20:42:18.685 DEBUG:teuthology.orchestra.run.smithi063:> true 2021-03-24T20:42:18.891 INFO:teuthology.run_tasks:Running task clock... 2021-03-24T20:42:18.902 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-03-24T20:42:18.903 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-24T20:42:18.905 DEBUG:teuthology.orchestra.run.smithi052:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-24T20:42:18.976 INFO:teuthology.orchestra.run.smithi052.stdout:24 Mar 20:42:18 ntpd[8506]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-03-24T20:42:18.977 INFO:teuthology.orchestra.run.smithi052.stdout:24 Mar 20:42:18 ntpd[8506]: Command line: ntpd -gq 2021-03-24T20:42:18.977 INFO:teuthology.orchestra.run.smithi052.stdout:24 Mar 20:42:18 ntpd[8506]: proto: precision = 0.123 usec (-23) 2021-03-24T20:42:18.978 INFO:teuthology.orchestra.run.smithi052.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-24T20:42:18.979 INFO:teuthology.orchestra.run.smithi052.stdout:24 Mar 20:42:18 ntpd[8506]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-24T20:42:18.979 INFO:teuthology.orchestra.run.smithi052.stderr:restrict ::: KOD does nothing without LIMITED. 2021-03-24T20:42:18.980 INFO:teuthology.orchestra.run.smithi052.stdout:24 Mar 20:42:18 ntpd[8506]: restrict ::: KOD does nothing without LIMITED. 2021-03-24T20:42:18.980 INFO:teuthology.orchestra.run.smithi052.stdout:24 Mar 20:42:18 ntpd[8506]: Listen and drop on 0 v6wildcard [::]:123 2021-03-24T20:42:18.981 INFO:teuthology.orchestra.run.smithi052.stdout:24 Mar 20:42:18 ntpd[8506]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-03-24T20:42:18.981 INFO:teuthology.orchestra.run.smithi052.stdout:24 Mar 20:42:18 ntpd[8506]: Listen normally on 2 lo 127.0.0.1:123 2021-03-24T20:42:18.981 INFO:teuthology.orchestra.run.smithi052.stdout:24 Mar 20:42:18 ntpd[8506]: Listen normally on 3 ens1f0 172.21.15.52:123 2021-03-24T20:42:18.981 INFO:teuthology.orchestra.run.smithi052.stdout:24 Mar 20:42:18 ntpd[8506]: Listen normally on 4 lo [::1]:123 2021-03-24T20:42:18.982 INFO:teuthology.orchestra.run.smithi052.stdout:24 Mar 20:42:18 ntpd[8506]: Listen normally on 5 ens1f0 [fe80::ec4:7aff:febd:15de%3]:123 2021-03-24T20:42:18.982 INFO:teuthology.orchestra.run.smithi052.stdout:24 Mar 20:42:18 ntpd[8506]: Listening on routing socket on fd #22 for interface updates 2021-03-24T20:42:19.978 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:19 ntpd[8506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:19.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:19 ntpd[8506]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:19.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:19 ntpd[8506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:19.980 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:19 ntpd[8506]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:21.978 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:21 ntpd[8506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:21.978 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:21 ntpd[8506]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:21.978 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:21 ntpd[8506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:21.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:21 ntpd[8506]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:21.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:21 ntpd[8506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:21.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:21 ntpd[8506]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:21.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:21 ntpd[8506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:21.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:21 ntpd[8506]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:22.978 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:22 ntpd[8506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:22.978 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:22 ntpd[8506]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:22.978 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:22 ntpd[8506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:22.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:22 ntpd[8506]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:22.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:22 ntpd[8506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:22.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:22 ntpd[8506]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:22.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:22 ntpd[8506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:22.980 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:22 ntpd[8506]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:23.978 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:23 ntpd[8506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:23.978 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:23 ntpd[8506]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:23.978 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:23 ntpd[8506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:23.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:23 ntpd[8506]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:23.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:23 ntpd[8506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:23.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:23 ntpd[8506]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:23.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:23 ntpd[8506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:23.980 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:23 ntpd[8506]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:24.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:24 ntpd[8506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:24.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:24 ntpd[8506]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:24.980 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:24 ntpd[8506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:24.980 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:24 ntpd[8506]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:24.981 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:24 ntpd[8506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:24.981 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:24 ntpd[8506]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:24.982 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:24 ntpd[8506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:24.982 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:24 ntpd[8506]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:25.978 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:25 ntpd[8506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:25.978 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:25 ntpd[8506]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:25.978 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:25 ntpd[8506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:25.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:25 ntpd[8506]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:25.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:25 ntpd[8506]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:25.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:25 ntpd[8506]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:25.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:25 ntpd[8506]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:25.979 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:25 ntpd[8506]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:25.980 INFO:teuthology.orchestra.run.smithi052.stdout:24 Mar 20:42:25 ntpd[8506]: ntpd: time slew -0.002699 s 2021-03-24T20:42:25.980 INFO:teuthology.orchestra.run.smithi052.stdout:ntpd: time slew -0.002699s 2021-03-24T20:42:25.981 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:25 ntpd[8506]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-03-24T20:42:25.981 INFO:teuthology.orchestra.run.smithi052.stderr:24 Mar 20:42:25 ntpd[8506]: can't open /var/log/ntpstats/loopstats.20210324: Permission denied 2021-03-24T20:42:26.032 INFO:teuthology.orchestra.run.smithi052.stdout: remote refid st t when poll reach delay offset jitter 2021-03-24T20:42:26.033 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================== 2021-03-24T20:42:26.033 INFO:teuthology.orchestra.run.smithi052.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-24T20:42:26.033 INFO:teuthology.orchestra.run.smithi052.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-24T20:42:26.034 INFO:teuthology.orchestra.run.smithi052.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-24T20:42:26.034 INFO:teuthology.orchestra.run.smithi052.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-24T20:42:26.035 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-03-24T20:42:26.035 DEBUG:teuthology.orchestra.run.smithi063:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-24T20:42:26.092 INFO:teuthology.orchestra.run.smithi063.stdout:24 Mar 20:42:25 ntpd[8558]: ntpd 4.2.8p10@1.3728-o (1): Starting 2021-03-24T20:42:26.093 INFO:teuthology.orchestra.run.smithi063.stdout:24 Mar 20:42:25 ntpd[8558]: Command line: ntpd -gq 2021-03-24T20:42:26.093 INFO:teuthology.orchestra.run.smithi063.stdout:24 Mar 20:42:25 ntpd[8558]: proto: precision = 0.127 usec (-23) 2021-03-24T20:42:26.094 INFO:teuthology.orchestra.run.smithi063.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-24T20:42:26.095 INFO:teuthology.orchestra.run.smithi063.stdout:24 Mar 20:42:25 ntpd[8558]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2021-03-24T20:42:26.095 INFO:teuthology.orchestra.run.smithi063.stderr:restrict ::: KOD does nothing without LIMITED. 2021-03-24T20:42:26.096 INFO:teuthology.orchestra.run.smithi063.stdout:24 Mar 20:42:25 ntpd[8558]: restrict ::: KOD does nothing without LIMITED. 2021-03-24T20:42:26.097 INFO:teuthology.orchestra.run.smithi063.stdout:24 Mar 20:42:25 ntpd[8558]: Listen and drop on 0 v6wildcard [::]:123 2021-03-24T20:42:26.097 INFO:teuthology.orchestra.run.smithi063.stdout:24 Mar 20:42:25 ntpd[8558]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2021-03-24T20:42:26.097 INFO:teuthology.orchestra.run.smithi063.stdout:24 Mar 20:42:25 ntpd[8558]: Listen normally on 2 lo 127.0.0.1:123 2021-03-24T20:42:26.097 INFO:teuthology.orchestra.run.smithi063.stdout:24 Mar 20:42:25 ntpd[8558]: Listen normally on 3 enp3s0f1 172.21.15.63:123 2021-03-24T20:42:26.098 INFO:teuthology.orchestra.run.smithi063.stdout:24 Mar 20:42:25 ntpd[8558]: Listen normally on 4 lo [::1]:123 2021-03-24T20:42:26.098 INFO:teuthology.orchestra.run.smithi063.stdout:24 Mar 20:42:25 ntpd[8558]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:f8d7%5]:123 2021-03-24T20:42:26.098 INFO:teuthology.orchestra.run.smithi063.stdout:24 Mar 20:42:25 ntpd[8558]: Listening on routing socket on fd #22 for interface updates 2021-03-24T20:42:27.094 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:26 ntpd[8558]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:27.094 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:26 ntpd[8558]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:27.094 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:26 ntpd[8558]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:27.095 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:26 ntpd[8558]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:27.095 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:26 ntpd[8558]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:27.095 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:26 ntpd[8558]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:27.095 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:26 ntpd[8558]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:27.095 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:26 ntpd[8558]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:29.092 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:28 ntpd[8558]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:29.093 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:28 ntpd[8558]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:29.093 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:28 ntpd[8558]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:29.093 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:28 ntpd[8558]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:29.093 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:28 ntpd[8558]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:29.094 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:28 ntpd[8558]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:29.094 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:28 ntpd[8558]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:29.094 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:28 ntpd[8558]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:30.092 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:29 ntpd[8558]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:30.092 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:29 ntpd[8558]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:30.092 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:29 ntpd[8558]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:30.093 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:29 ntpd[8558]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:30.093 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:29 ntpd[8558]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:30.093 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:29 ntpd[8558]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:30.093 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:29 ntpd[8558]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:30.093 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:29 ntpd[8558]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:31.091 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:30 ntpd[8558]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:31.092 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:30 ntpd[8558]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:31.092 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:30 ntpd[8558]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:31.092 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:30 ntpd[8558]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:31.092 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:30 ntpd[8558]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:31.092 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:30 ntpd[8558]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:31.093 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:30 ntpd[8558]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:31.093 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:30 ntpd[8558]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:32.091 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:31 ntpd[8558]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:32.091 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:31 ntpd[8558]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:32.091 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:31 ntpd[8558]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:32.092 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:31 ntpd[8558]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:32.092 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:31 ntpd[8558]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:32.092 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:31 ntpd[8558]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:32.093 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:31 ntpd[8558]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:32.093 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:31 ntpd[8558]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:33.090 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:32 ntpd[8558]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2021-03-24T20:42:33.090 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:32 ntpd[8558]: can't open /var/log/ntpstats/rawstats.20210324: Permission denied 2021-03-24T20:42:33.091 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:32 ntpd[8558]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2021-03-24T20:42:33.091 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:32 ntpd[8558]: can't open /var/log/ntpstats/peerstats.20210324: Permission denied 2021-03-24T20:42:33.091 INFO:teuthology.orchestra.run.smithi063.stdout:24 Mar 20:42:33 ntpd[8558]: ntpd: time set +0.355822 s 2021-03-24T20:42:33.092 INFO:teuthology.orchestra.run.smithi063.stdout:ntpd: time set +0.355822s 2021-03-24T20:42:33.092 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:33 ntpd[8558]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2021-03-24T20:42:33.092 INFO:teuthology.orchestra.run.smithi063.stderr:24 Mar 20:42:33 ntpd[8558]: can't open /var/log/ntpstats/loopstats.20210324: Permission denied 2021-03-24T20:42:33.144 INFO:teuthology.orchestra.run.smithi063.stdout: remote refid st t when poll reach delay offset jitter 2021-03-24T20:42:33.145 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================== 2021-03-24T20:42:33.145 INFO:teuthology.orchestra.run.smithi063.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-24T20:42:33.145 INFO:teuthology.orchestra.run.smithi063.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-24T20:42:33.145 INFO:teuthology.orchestra.run.smithi063.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-24T20:42:33.146 INFO:teuthology.orchestra.run.smithi063.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2021-03-24T20:42:33.146 INFO:teuthology.run_tasks:Running task install... 2021-03-24T20:42:33.161 DEBUG:teuthology.task.install:project ceph 2021-03-24T20:42:33.169 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'branch': 'octopus', 'sha1': '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2'}} 2021-03-24T20:42:33.169 DEBUG:teuthology.task.install:config {'branch': 'octopus', 'sha1': '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2'} 2021-03-24T20:42:33.169 INFO:teuthology.task.install:Using flavor: basic 2021-03-24T20:42:33.181 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-03-24T20:42:33.182 INFO:teuthology.task.install:extra packages: [] 2021-03-24T20:42:33.183 DEBUG:teuthology.orchestra.run.smithi052:> sudo lsb_release -is 2021-03-24T20:42:33.238 INFO:teuthology.orchestra.run.smithi052.stdout:Ubuntu 2021-03-24T20:42:33.244 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-24T20:42:33.245 DEBUG:teuthology.orchestra.run.smithi063:> sudo lsb_release -is 2021-03-24T20:42:33.246 DEBUG:teuthology.orchestra.run.smithi052:> sudo apt-key list | grep Ceph 2021-03-24T20:42:33.299 INFO:teuthology.orchestra.run.smithi063.stdout:Ubuntu 2021-03-24T20:42:33.305 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-24T20:42:33.306 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-key list | grep Ceph 2021-03-24T20:42:33.347 INFO:teuthology.orchestra.run.smithi052.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-24T20:42:33.395 INFO:teuthology.orchestra.run.smithi063.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-24T20:42:33.407 INFO:teuthology.orchestra.run.smithi052.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-03-24T20:42:33.407 INFO:teuthology.orchestra.run.smithi052.stdout:uid [ unknown] Ceph.com (release key) 2021-03-24T20:42:33.409 DEBUG:teuthology.orchestra.run.smithi052:> sudo lsb_release -is 2021-03-24T20:42:33.450 INFO:teuthology.orchestra.run.smithi063.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-03-24T20:42:33.450 INFO:teuthology.orchestra.run.smithi063.stdout:uid [ unknown] Ceph.com (release key) 2021-03-24T20:42:33.452 DEBUG:teuthology.orchestra.run.smithi063:> sudo lsb_release -is 2021-03-24T20:42:33.464 INFO:teuthology.orchestra.run.smithi052.stdout:Ubuntu 2021-03-24T20:42:33.469 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-24T20:42:33.469 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-03-24T20:42:33.470 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-03-24T20:42:33.470 INFO:teuthology.packaging:ref: None 2021-03-24T20:42:33.470 INFO:teuthology.packaging:tag: None 2021-03-24T20:42:33.471 INFO:teuthology.packaging:branch: octopus 2021-03-24T20:42:33.471 INFO:teuthology.packaging:sha1: 9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2 2021-03-24T20:42:33.472 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=octopus 2021-03-24T20:42:33.509 INFO:teuthology.orchestra.run.smithi063.stdout:Ubuntu 2021-03-24T20:42:33.511 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-24T20:42:33.511 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse on remote deb x86_64 2021-03-24T20:42:33.511 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-03-24T20:42:33.512 INFO:teuthology.packaging:ref: None 2021-03-24T20:42:33.512 INFO:teuthology.packaging:tag: None 2021-03-24T20:42:33.512 INFO:teuthology.packaging:branch: octopus 2021-03-24T20:42:33.513 INFO:teuthology.packaging:sha1: 9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2 2021-03-24T20:42:33.513 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=octopus 2021-03-24T20:42:33.694 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default/ 2021-03-24T20:42:33.694 INFO:teuthology.task.install.deb:Package version is 15.2.10-76-gdcf77bd9-1bionic 2021-03-24T20:42:33.732 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default/ 2021-03-24T20:42:33.733 INFO:teuthology.task.install.deb:Package version is 15.2.10-76-gdcf77bd9-1bionic 2021-03-24T20:42:33.802 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:42:33.802 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-03-24T20:42:33.818 DEBUG:teuthology.orchestra.run.smithi052:> sudo apt-get update 2021-03-24T20:42:33.841 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:42:33.841 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-03-24T20:42:33.857 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-get update 2021-03-24T20:42:34.006 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-03-24T20:42:34.036 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-03-24T20:42:34.062 INFO:teuthology.orchestra.run.smithi052.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic InRelease 2021-03-24T20:42:34.083 INFO:teuthology.orchestra.run.smithi052.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic Release [7,690 B] 2021-03-24T20:42:34.092 INFO:teuthology.orchestra.run.smithi063.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic InRelease 2021-03-24T20:42:34.105 INFO:teuthology.orchestra.run.smithi052.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic Release.gpg 2021-03-24T20:42:34.113 INFO:teuthology.orchestra.run.smithi063.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic Release [7,690 B] 2021-03-24T20:42:34.122 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:5 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-24T20:42:34.126 INFO:teuthology.orchestra.run.smithi052.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,081 B] 2021-03-24T20:42:34.135 INFO:teuthology.orchestra.run.smithi063.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic Release.gpg 2021-03-24T20:42:34.157 INFO:teuthology.orchestra.run.smithi063.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.8 kB] 2021-03-24T20:42:34.161 INFO:teuthology.orchestra.run.smithi052.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 Packages [15.8 kB] 2021-03-24T20:42:34.193 INFO:teuthology.orchestra.run.smithi063.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main i386 Packages [3,081 B] 2021-03-24T20:42:34.201 INFO:teuthology.orchestra.run.smithi052.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-03-24T20:42:34.285 INFO:teuthology.orchestra.run.smithi052.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-03-24T20:42:34.670 INFO:teuthology.orchestra.run.smithi052.stdout:Fetched 101 kB in 1s (158 kB/s) 2021-03-24T20:42:35.302 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:7 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-24T20:42:35.382 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:8 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-03-24T20:42:35.466 INFO:teuthology.orchestra.run.smithi063.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-03-24T20:42:35.946 INFO:teuthology.orchestra.run.smithi063.stdout:Fetched 101 kB in 2s (53.5 kB/s) 2021-03-24T20:42:35.979 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2021-03-24T20:42:36.027 DEBUG:teuthology.orchestra.run.smithi052:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.10-76-gdcf77bd9-1bionic cephadm=15.2.10-76-gdcf77bd9-1bionic ceph-mds=15.2.10-76-gdcf77bd9-1bionic ceph-mgr=15.2.10-76-gdcf77bd9-1bionic ceph-common=15.2.10-76-gdcf77bd9-1bionic ceph-fuse=15.2.10-76-gdcf77bd9-1bionic ceph-test=15.2.10-76-gdcf77bd9-1bionic radosgw=15.2.10-76-gdcf77bd9-1bionic python3-rados=15.2.10-76-gdcf77bd9-1bionic python3-rgw=15.2.10-76-gdcf77bd9-1bionic python3-cephfs=15.2.10-76-gdcf77bd9-1bionic python3-rbd=15.2.10-76-gdcf77bd9-1bionic libcephfs2=15.2.10-76-gdcf77bd9-1bionic libcephfs-dev=15.2.10-76-gdcf77bd9-1bionic librados2=15.2.10-76-gdcf77bd9-1bionic librbd1=15.2.10-76-gdcf77bd9-1bionic rbd-fuse=15.2.10-76-gdcf77bd9-1bionic 2021-03-24T20:42:36.113 INFO:teuthology.orchestra.run.smithi052.stdout:Reading package lists... 2021-03-24T20:42:36.396 INFO:teuthology.orchestra.run.smithi052.stdout:Building dependency tree... 2021-03-24T20:42:36.397 INFO:teuthology.orchestra.run.smithi052.stdout:Reading state information... 2021-03-24T20:42:36.589 INFO:teuthology.orchestra.run.smithi052.stdout:The following additional packages will be installed: 2021-03-24T20:42:36.589 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2021-03-24T20:42:36.590 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2021-03-24T20:42:36.591 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2021-03-24T20:42:36.592 INFO:teuthology.orchestra.run.smithi052.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-03-24T20:42:36.592 INFO:teuthology.orchestra.run.smithi052.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-03-24T20:42:36.592 INFO:teuthology.orchestra.run.smithi052.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-03-24T20:42:36.593 INFO:teuthology.orchestra.run.smithi052.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-24T20:42:36.593 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-03-24T20:42:36.594 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-03-24T20:42:36.594 INFO:teuthology.orchestra.run.smithi052.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-03-24T20:42:36.595 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-03-24T20:42:36.595 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-03-24T20:42:36.595 INFO:teuthology.orchestra.run.smithi052.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-03-24T20:42:36.596 INFO:teuthology.orchestra.run.smithi052.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-03-24T20:42:36.596 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-03-24T20:42:36.596 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-03-24T20:42:36.596 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-03-24T20:42:36.597 INFO:teuthology.orchestra.run.smithi052.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-03-24T20:42:36.597 INFO:teuthology.orchestra.run.smithi052.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-03-24T20:42:36.597 INFO:teuthology.orchestra.run.smithi052.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-03-24T20:42:36.597 INFO:teuthology.orchestra.run.smithi052.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-03-24T20:42:36.597 INFO:teuthology.orchestra.run.smithi052.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-03-24T20:42:36.598 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-03-24T20:42:36.598 INFO:teuthology.orchestra.run.smithi052.stdout:Suggested packages: 2021-03-24T20:42:36.599 INFO:teuthology.orchestra.run.smithi052.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-03-24T20:42:36.599 INFO:teuthology.orchestra.run.smithi052.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-03-24T20:42:36.599 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-03-24T20:42:36.599 INFO:teuthology.orchestra.run.smithi052.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-03-24T20:42:36.600 INFO:teuthology.orchestra.run.smithi052.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-03-24T20:42:36.600 INFO:teuthology.orchestra.run.smithi052.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-03-24T20:42:36.600 INFO:teuthology.orchestra.run.smithi052.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-03-24T20:42:36.600 INFO:teuthology.orchestra.run.smithi052.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-03-24T20:42:36.601 INFO:teuthology.orchestra.run.smithi052.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-03-24T20:42:36.601 INFO:teuthology.orchestra.run.smithi052.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-03-24T20:42:36.601 INFO:teuthology.orchestra.run.smithi052.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-03-24T20:42:36.601 INFO:teuthology.orchestra.run.smithi052.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-03-24T20:42:36.601 INFO:teuthology.orchestra.run.smithi052.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-03-24T20:42:36.602 INFO:teuthology.orchestra.run.smithi052.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-03-24T20:42:36.602 INFO:teuthology.orchestra.run.smithi052.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-03-24T20:42:36.699 INFO:teuthology.orchestra.run.smithi052.stdout:The following NEW packages will be installed: 2021-03-24T20:42:36.699 INFO:teuthology.orchestra.run.smithi052.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-03-24T20:42:36.700 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2021-03-24T20:42:36.700 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2021-03-24T20:42:36.700 INFO:teuthology.orchestra.run.smithi052.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2021-03-24T20:42:36.702 INFO:teuthology.orchestra.run.smithi052.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-03-24T20:42:36.702 INFO:teuthology.orchestra.run.smithi052.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-03-24T20:42:36.702 INFO:teuthology.orchestra.run.smithi052.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-03-24T20:42:36.703 INFO:teuthology.orchestra.run.smithi052.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-24T20:42:36.703 INFO:teuthology.orchestra.run.smithi052.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-03-24T20:42:36.703 INFO:teuthology.orchestra.run.smithi052.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-03-24T20:42:36.704 INFO:teuthology.orchestra.run.smithi052.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-03-24T20:42:36.704 INFO:teuthology.orchestra.run.smithi052.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-03-24T20:42:36.704 INFO:teuthology.orchestra.run.smithi052.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-03-24T20:42:36.704 INFO:teuthology.orchestra.run.smithi052.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-03-24T20:42:36.705 INFO:teuthology.orchestra.run.smithi052.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-03-24T20:42:36.705 INFO:teuthology.orchestra.run.smithi052.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-03-24T20:42:36.705 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-03-24T20:42:36.705 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-03-24T20:42:36.706 INFO:teuthology.orchestra.run.smithi052.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-03-24T20:42:36.706 INFO:teuthology.orchestra.run.smithi052.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-03-24T20:42:36.706 INFO:teuthology.orchestra.run.smithi052.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-03-24T20:42:36.706 INFO:teuthology.orchestra.run.smithi052.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-03-24T20:42:36.707 INFO:teuthology.orchestra.run.smithi052.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-03-24T20:42:36.707 INFO:teuthology.orchestra.run.smithi052.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-03-24T20:42:36.707 INFO:teuthology.orchestra.run.smithi052.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-03-24T20:42:36.708 INFO:teuthology.orchestra.run.smithi052.stdout:The following packages will be upgraded: 2021-03-24T20:42:36.708 INFO:teuthology.orchestra.run.smithi052.stdout: librados2 librbd1 2021-03-24T20:42:36.855 INFO:teuthology.orchestra.run.smithi052.stdout:2 upgraded, 111 newly installed, 0 to remove and 29 not upgraded. 2021-03-24T20:42:36.855 INFO:teuthology.orchestra.run.smithi052.stdout:Need to get 156 MB of archives. 2021-03-24T20:42:36.856 INFO:teuthology.orchestra.run.smithi052.stdout:After this operation, 725 MB of additional disk space will be used. 2021-03-24T20:42:36.856 INFO:teuthology.orchestra.run.smithi052.stdout:Get:1 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.2.10-76-gdcf77bd9-1bionic [1,639 kB] 2021-03-24T20:42:36.895 INFO:teuthology.orchestra.run.smithi052.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-03-24T20:42:36.977 INFO:teuthology.orchestra.run.smithi052.stdout:Get:3 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-03-24T20:42:37.027 INFO:teuthology.orchestra.run.smithi052.stdout:Get:4 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.2.10-76-gdcf77bd9-1bionic [3,060 kB] 2021-03-24T20:42:37.121 INFO:teuthology.orchestra.run.smithi052.stdout:Get:5 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.2.10-76-gdcf77bd9-1bionic [435 kB] 2021-03-24T20:42:37.243 INFO:teuthology.orchestra.run.smithi052.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.2.10-76-gdcf77bd9-1bionic [345 kB] 2021-03-24T20:42:37.244 INFO:teuthology.orchestra.run.smithi052.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.2.10-76-gdcf77bd9-1bionic [20.7 kB] 2021-03-24T20:42:37.244 INFO:teuthology.orchestra.run.smithi052.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.2.10-76-gdcf77bd9-1bionic [133 kB] 2021-03-24T20:42:37.244 INFO:teuthology.orchestra.run.smithi052.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 15.2.10-76-gdcf77bd9-1bionic [29.1 kB] 2021-03-24T20:42:37.245 INFO:teuthology.orchestra.run.smithi052.stdout:Get:10 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 15.2.10-76-gdcf77bd9-1bionic [326 kB] 2021-03-24T20:42:37.245 INFO:teuthology.orchestra.run.smithi052.stdout:Get:11 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.2.10-76-gdcf77bd9-1bionic [2,902 kB] 2021-03-24T20:42:37.245 INFO:teuthology.orchestra.run.smithi052.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-03-24T20:42:37.245 INFO:teuthology.orchestra.run.smithi052.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-03-24T20:42:37.246 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2021-03-24T20:42:37.246 INFO:teuthology.orchestra.run.smithi052.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-03-24T20:42:37.257 INFO:teuthology.orchestra.run.smithi052.stdout:Get:15 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 15.2.10-76-gdcf77bd9-1bionic [99.4 kB] 2021-03-24T20:42:37.260 INFO:teuthology.orchestra.run.smithi052.stdout:Get:16 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.2.10-76-gdcf77bd9-1bionic [338 kB] 2021-03-24T20:42:37.271 INFO:teuthology.orchestra.run.smithi052.stdout:Get:17 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.2.10-76-gdcf77bd9-1bionic [18.8 MB] 2021-03-24T20:42:37.293 DEBUG:teuthology.orchestra.run.smithi063:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.10-76-gdcf77bd9-1bionic cephadm=15.2.10-76-gdcf77bd9-1bionic ceph-mds=15.2.10-76-gdcf77bd9-1bionic ceph-mgr=15.2.10-76-gdcf77bd9-1bionic ceph-common=15.2.10-76-gdcf77bd9-1bionic ceph-fuse=15.2.10-76-gdcf77bd9-1bionic ceph-test=15.2.10-76-gdcf77bd9-1bionic radosgw=15.2.10-76-gdcf77bd9-1bionic python3-rados=15.2.10-76-gdcf77bd9-1bionic python3-rgw=15.2.10-76-gdcf77bd9-1bionic python3-cephfs=15.2.10-76-gdcf77bd9-1bionic python3-rbd=15.2.10-76-gdcf77bd9-1bionic libcephfs2=15.2.10-76-gdcf77bd9-1bionic libcephfs-dev=15.2.10-76-gdcf77bd9-1bionic librados2=15.2.10-76-gdcf77bd9-1bionic librbd1=15.2.10-76-gdcf77bd9-1bionic rbd-fuse=15.2.10-76-gdcf77bd9-1bionic 2021-03-24T20:42:37.342 INFO:teuthology.orchestra.run.smithi052.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-03-24T20:42:37.345 INFO:teuthology.orchestra.run.smithi052.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-03-24T20:42:37.374 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2021-03-24T20:42:37.387 INFO:teuthology.orchestra.run.smithi052.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-03-24T20:42:37.394 INFO:teuthology.orchestra.run.smithi052.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-03-24T20:42:37.403 INFO:teuthology.orchestra.run.smithi052.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-03-24T20:42:37.413 INFO:teuthology.orchestra.run.smithi052.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-03-24T20:42:37.420 INFO:teuthology.orchestra.run.smithi052.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-03-24T20:42:37.440 INFO:teuthology.orchestra.run.smithi052.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-03-24T20:42:37.443 INFO:teuthology.orchestra.run.smithi052.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-03-24T20:42:37.472 INFO:teuthology.orchestra.run.smithi052.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-03-24T20:42:37.481 INFO:teuthology.orchestra.run.smithi052.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-03-24T20:42:37.483 INFO:teuthology.orchestra.run.smithi052.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-03-24T20:42:37.552 INFO:teuthology.orchestra.run.smithi052.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-03-24T20:42:37.609 INFO:teuthology.orchestra.run.smithi052.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-03-24T20:42:37.610 INFO:teuthology.orchestra.run.smithi052.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-03-24T20:42:37.610 INFO:teuthology.orchestra.run.smithi052.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-03-24T20:42:37.610 INFO:teuthology.orchestra.run.smithi052.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-03-24T20:42:37.611 INFO:teuthology.orchestra.run.smithi052.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-03-24T20:42:37.611 INFO:teuthology.orchestra.run.smithi052.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-03-24T20:42:37.632 INFO:teuthology.orchestra.run.smithi052.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-03-24T20:42:37.632 INFO:teuthology.orchestra.run.smithi052.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-03-24T20:42:37.636 INFO:teuthology.orchestra.run.smithi063.stdout:Building dependency tree... 2021-03-24T20:42:37.637 INFO:teuthology.orchestra.run.smithi063.stdout:Reading state information... 2021-03-24T20:42:37.653 INFO:teuthology.orchestra.run.smithi052.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-03-24T20:42:37.712 INFO:teuthology.orchestra.run.smithi052.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-03-24T20:42:37.712 INFO:teuthology.orchestra.run.smithi052.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-03-24T20:42:37.713 INFO:teuthology.orchestra.run.smithi052.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-03-24T20:42:37.717 INFO:teuthology.orchestra.run.smithi052.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-03-24T20:42:37.725 INFO:teuthology.orchestra.run.smithi052.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-03-24T20:42:37.729 INFO:teuthology.orchestra.run.smithi052.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-03-24T20:42:37.730 INFO:teuthology.orchestra.run.smithi052.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-03-24T20:42:37.794 INFO:teuthology.orchestra.run.smithi052.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-03-24T20:42:37.818 INFO:teuthology.orchestra.run.smithi063.stdout:The following additional packages will be installed: 2021-03-24T20:42:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2021-03-24T20:42:37.818 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2021-03-24T20:42:37.819 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook ceph-mon ceph-osd jq libjq1 libleveldb1v5 liblttng-ust0 2021-03-24T20:42:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: liboath0 libonig4 librabbitmq4 libradosstriper1 librdkafka1 librgw2 2021-03-24T20:42:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: libsnappy1v5 libxslt1.1 nvme-cli python-attr python-funcsigs 2021-03-24T20:42:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: python-pastedeploy-tpl python-pluggy python-py python-pytest 2021-03-24T20:42:37.820 INFO:teuthology.orchestra.run.smithi063.stdout: python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-24T20:42:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse python3-ceph-common python3-cffi-backend 2021-03-24T20:42:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cherrypy3 python3-cryptography python3-cycler python3-dateutil 2021-03-24T20:42:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: python3-decorator python3-fasteners python3-html5lib python3-httplib2 2021-03-24T20:42:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jinja2 python3-joblib python3-json-pointer python3-jsonpatch 2021-03-24T20:42:37.821 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2021-03-24T20:42:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: python3-markupsafe python3-matplotlib python3-monotonic python3-numpy 2021-03-24T20:42:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: python3-oauth2client python3-olefile python3-openssl python3-paste 2021-03-24T20:42:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pastedeploy python3-pastescript python3-pecan python3-pil 2021-03-24T20:42:37.822 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pluggy python3-prettytable python3-py python3-pyasn1 2021-03-24T20:42:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1-modules python3-pyinotify python3-pyparsing python3-pytest 2021-03-24T20:42:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: python3-repoze.lru python3-routes python3-rsa python3-scipy 2021-03-24T20:42:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: python3-setuptools python3-simplegeneric python3-simplejson 2021-03-24T20:42:37.823 INFO:teuthology.orchestra.run.smithi063.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib 2021-03-24T20:42:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: python3-sqlalchemy python3-tempita python3-tk python3-tz python3-uritemplate 2021-03-24T20:42:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: python3-waitress python3-webencodings python3-webob python3-websocket 2021-03-24T20:42:37.824 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webtest python3-werkzeug socat xmlstarlet 2021-03-24T20:42:37.825 INFO:teuthology.orchestra.run.smithi063.stdout:Suggested packages: 2021-03-24T20:42:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: python3-influxdb python-attr-doc python-funcsigs-doc subversion 2021-03-24T20:42:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: python-pytest-xdist python-mock python-cryptography-doc 2021-03-24T20:42:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cryptography-vectors python-cycler-doc python3-genshi 2021-03-24T20:42:37.825 INFO:teuthology.orchestra.run.smithi063.stdout: python-jinja2-doc python3-crypto python3-lxml-dbg python-lxml-doc 2021-03-24T20:42:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: python3-beaker python-mako-doc dvipng ffmpeg gir1.2-gtk-3.0 ghostscript 2021-03-24T20:42:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: inkscape ipython3 python-matplotlib-doc python3-cairocffi python3-gi-cairo 2021-03-24T20:42:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: python3-gobject python3-pyqt4 python3-sip python3-tornado 2021-03-24T20:42:37.826 INFO:teuthology.orchestra.run.smithi063.stdout: texlive-extra-utils texlive-latex-extra ttf-staypuft python-numpy-doc 2021-03-24T20:42:37.827 INFO:teuthology.orchestra.run.smithi063.stdout: python3-dev python3-numpy-dbg python-openssl-doc python3-openssl-dbg 2021-03-24T20:42:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2021-03-24T20:42:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: python-pecan-doc python-pil-doc python3-pil-dbg python-pyinotify-doc 2021-03-24T20:42:37.828 INFO:teuthology.orchestra.run.smithi063.stdout: python-pyparsing-doc python-scipy-doc python-setuptools-doc python3-dap 2021-03-24T20:42:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: python-sklearn-doc python-sqlalchemy-doc python3-psycopg2 python3-pymysql 2021-03-24T20:42:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: python3-fdb tix python3-tk-dbg python-waitress-doc python-webob-doc 2021-03-24T20:42:37.829 INFO:teuthology.orchestra.run.smithi063.stdout: python-webtest-doc python3-termcolor python3-watchdog python-werkzeug-doc 2021-03-24T20:42:37.879 INFO:teuthology.orchestra.run.smithi052.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-03-24T20:42:38.033 INFO:teuthology.orchestra.run.smithi052.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-03-24T20:42:38.035 INFO:teuthology.orchestra.run.smithi052.stdout:Get:50 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.2.10-76-gdcf77bd9-1bionic [5,176 kB] 2021-03-24T20:42:38.036 INFO:teuthology.orchestra.run.smithi063.stdout:The following NEW packages will be installed: 2021-03-24T20:42:38.036 INFO:teuthology.orchestra.run.smithi063.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2021-03-24T20:42:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2021-03-24T20:42:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2021-03-24T20:42:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: ceph-mgr-rook ceph-mon ceph-osd ceph-test cephadm jq libcephfs-dev 2021-03-24T20:42:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: libcephfs2 libjq1 libleveldb1v5 liblttng-ust0 liboath0 libonig4 librabbitmq4 2021-03-24T20:42:38.037 INFO:teuthology.orchestra.run.smithi063.stdout: libradosstriper1 librdkafka1 librgw2 libsnappy1v5 libxslt1.1 nvme-cli 2021-03-24T20:42:38.038 INFO:teuthology.orchestra.run.smithi063.stdout: python-attr python-funcsigs python-pastedeploy-tpl python-pluggy python-py 2021-03-24T20:42:38.038 INFO:teuthology.orchestra.run.smithi063.stdout: python-pytest python3-asn1crypto python3-attr python3-bcrypt python3-bs4 2021-03-24T20:42:38.038 INFO:teuthology.orchestra.run.smithi063.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs 2021-03-24T20:42:38.038 INFO:teuthology.orchestra.run.smithi063.stdout: python3-cffi-backend python3-cherrypy3 python3-cryptography python3-cycler 2021-03-24T20:42:38.038 INFO:teuthology.orchestra.run.smithi063.stdout: python3-dateutil python3-decorator python3-fasteners python3-html5lib 2021-03-24T20:42:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: python3-httplib2 python3-jinja2 python3-joblib python3-json-pointer 2021-03-24T20:42:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: python3-jsonpatch python3-jwt python3-kubernetes python3-logutils 2021-03-24T20:42:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: python3-lxml python3-mako python3-markupsafe python3-matplotlib 2021-03-24T20:42:38.039 INFO:teuthology.orchestra.run.smithi063.stdout: python3-monotonic python3-numpy python3-oauth2client python3-olefile 2021-03-24T20:42:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2021-03-24T20:42:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pecan python3-pil python3-pluggy python3-prettytable python3-py 2021-03-24T20:42:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyparsing 2021-03-24T20:42:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: python3-pytest python3-rados python3-rbd python3-repoze.lru python3-rgw 2021-03-24T20:42:38.040 INFO:teuthology.orchestra.run.smithi063.stdout: python3-routes python3-rsa python3-scipy python3-setuptools 2021-03-24T20:42:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2021-03-24T20:42:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: python3-sklearn python3-sklearn-lib python3-sqlalchemy python3-tempita 2021-03-24T20:42:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: python3-tk python3-tz python3-uritemplate python3-waitress 2021-03-24T20:42:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: python3-webencodings python3-webob python3-websocket python3-webtest 2021-03-24T20:42:38.041 INFO:teuthology.orchestra.run.smithi063.stdout: python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2021-03-24T20:42:38.042 INFO:teuthology.orchestra.run.smithi063.stdout:The following packages will be upgraded: 2021-03-24T20:42:38.042 INFO:teuthology.orchestra.run.smithi063.stdout: librados2 librbd1 2021-03-24T20:42:38.059 INFO:teuthology.orchestra.run.smithi052.stdout:Get:51 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 15.2.10-76-gdcf77bd9-1bionic [161 kB] 2021-03-24T20:42:38.063 INFO:teuthology.orchestra.run.smithi052.stdout:Get:52 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.2.10-76-gdcf77bd9-1bionic [1,284 kB] 2021-03-24T20:42:38.106 INFO:teuthology.orchestra.run.smithi052.stdout:Get:53 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.2.10-76-gdcf77bd9-1bionic [5,943 kB] 2021-03-24T20:42:38.123 INFO:teuthology.orchestra.run.smithi063.stdout:2 upgraded, 111 newly installed, 0 to remove and 29 not upgraded. 2021-03-24T20:42:38.123 INFO:teuthology.orchestra.run.smithi063.stdout:Need to get 156 MB of archives. 2021-03-24T20:42:38.123 INFO:teuthology.orchestra.run.smithi063.stdout:After this operation, 725 MB of additional disk space will be used. 2021-03-24T20:42:38.124 INFO:teuthology.orchestra.run.smithi063.stdout:Get:1 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-uritemplate all 0.6-1ubuntu1 [5,126 B] 2021-03-24T20:42:38.153 INFO:teuthology.orchestra.run.smithi052.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-03-24T20:42:38.204 INFO:teuthology.orchestra.run.smithi063.stdout:Get:2 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liblttng-ust0 amd64 2.10.1-1 [154 kB] 2021-03-24T20:42:38.204 INFO:teuthology.orchestra.run.smithi052.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-03-24T20:42:38.206 INFO:teuthology.orchestra.run.smithi052.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-03-24T20:42:38.236 INFO:teuthology.orchestra.run.smithi052.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-03-24T20:42:38.325 INFO:teuthology.orchestra.run.smithi052.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-03-24T20:42:38.327 INFO:teuthology.orchestra.run.smithi052.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.3 [28.8 kB] 2021-03-24T20:42:38.327 INFO:teuthology.orchestra.run.smithi052.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-03-24T20:42:38.328 INFO:teuthology.orchestra.run.smithi052.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-03-24T20:42:38.328 INFO:teuthology.orchestra.run.smithi052.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-03-24T20:42:38.329 INFO:teuthology.orchestra.run.smithi052.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-03-24T20:42:38.329 INFO:teuthology.orchestra.run.smithi052.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-03-24T20:42:38.329 INFO:teuthology.orchestra.run.smithi052.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-03-24T20:42:38.347 INFO:teuthology.orchestra.run.smithi063.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 librbd1 amd64 15.2.10-76-gdcf77bd9-1bionic [1,639 kB] 2021-03-24T20:42:38.396 INFO:teuthology.orchestra.run.smithi052.stdout:Get:66 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.2.10-76-gdcf77bd9-1bionic [22.8 MB] 2021-03-24T20:42:38.397 INFO:teuthology.orchestra.run.smithi052.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-03-24T20:42:38.397 INFO:teuthology.orchestra.run.smithi052.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-03-24T20:42:38.402 INFO:teuthology.orchestra.run.smithi052.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-03-24T20:42:38.403 INFO:teuthology.orchestra.run.smithi052.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-03-24T20:42:38.404 INFO:teuthology.orchestra.run.smithi052.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-03-24T20:42:38.409 INFO:teuthology.orchestra.run.smithi052.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-03-24T20:42:38.410 INFO:teuthology.orchestra.run.smithi052.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-03-24T20:42:38.410 INFO:teuthology.orchestra.run.smithi052.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-03-24T20:42:38.461 INFO:teuthology.orchestra.run.smithi063.stdout:Get:4 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-prettytable all 0.7.2-3 [19.7 kB] 2021-03-24T20:42:38.467 INFO:teuthology.orchestra.run.smithi063.stdout:Get:5 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 librabbitmq4 amd64 0.8.0-1ubuntu0.18.04.2 [33.9 kB] 2021-03-24T20:42:38.473 INFO:teuthology.orchestra.run.smithi063.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic/universe amd64 librdkafka1 amd64 0.11.3-1build1 [293 kB] 2021-03-24T20:42:38.475 INFO:teuthology.orchestra.run.smithi052.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-03-24T20:42:38.479 INFO:teuthology.orchestra.run.smithi052.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-03-24T20:42:38.555 INFO:teuthology.orchestra.run.smithi052.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-03-24T20:42:38.563 INFO:teuthology.orchestra.run.smithi052.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-03-24T20:42:38.564 INFO:teuthology.orchestra.run.smithi052.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-03-24T20:42:38.564 INFO:teuthology.orchestra.run.smithi052.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-03-24T20:42:38.564 INFO:teuthology.orchestra.run.smithi052.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-03-24T20:42:38.566 INFO:teuthology.orchestra.run.smithi052.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-03-24T20:42:38.566 INFO:teuthology.orchestra.run.smithi052.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-03-24T20:42:38.567 INFO:teuthology.orchestra.run.smithi052.stdout:Get:84 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-03-24T20:42:38.580 INFO:teuthology.orchestra.run.smithi063.stdout:Get:7 http://archive.ubuntu.com/ubuntu bionic/main amd64 libsnappy1v5 amd64 1.1.7-1 [16.0 kB] 2021-03-24T20:42:38.586 INFO:teuthology.orchestra.run.smithi063.stdout:Get:8 http://archive.ubuntu.com/ubuntu bionic/main amd64 libleveldb1v5 amd64 1.20-2 [136 kB] 2021-03-24T20:42:38.619 INFO:teuthology.orchestra.run.smithi063.stdout:Get:9 http://archive.ubuntu.com/ubuntu bionic/universe amd64 liboath0 amd64 2.6.1-1 [44.7 kB] 2021-03-24T20:42:38.626 INFO:teuthology.orchestra.run.smithi063.stdout:Get:10 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-dateutil all 2.6.1-1 [52.3 kB] 2021-03-24T20:42:38.634 INFO:teuthology.orchestra.run.smithi063.stdout:Get:11 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-asn1crypto all 0.24.0-1 [72.8 kB] 2021-03-24T20:42:38.635 INFO:teuthology.orchestra.run.smithi052.stdout:Get:85 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-03-24T20:42:38.636 INFO:teuthology.orchestra.run.smithi052.stdout:Get:86 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-03-24T20:42:38.647 INFO:teuthology.orchestra.run.smithi063.stdout:Get:12 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-cffi-backend amd64 1.11.5-1 [64.6 kB] 2021-03-24T20:42:38.652 INFO:teuthology.orchestra.run.smithi063.stdout:Get:13 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-cryptography amd64 2.1.4-1ubuntu1.4 [220 kB] 2021-03-24T20:42:38.675 INFO:teuthology.orchestra.run.smithi063.stdout:Get:14 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-openssl all 17.5.0-1ubuntu1 [41.5 kB] 2021-03-24T20:42:38.682 INFO:teuthology.orchestra.run.smithi063.stdout:Get:15 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-bcrypt amd64 3.1.4-2 [29.9 kB] 2021-03-24T20:42:38.710 INFO:teuthology.orchestra.run.smithi063.stdout:Get:16 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cherrypy3 all 8.9.1-2 [160 kB] 2021-03-24T20:42:38.715 INFO:teuthology.orchestra.run.smithi052.stdout:Get:87 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-03-24T20:42:38.722 INFO:teuthology.orchestra.run.smithi063.stdout:Get:17 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jwt all 1.5.3+ds1-1 [15.9 kB] 2021-03-24T20:42:38.722 INFO:teuthology.orchestra.run.smithi063.stdout:Get:18 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-logutils all 0.3.3-5 [16.7 kB] 2021-03-24T20:42:38.724 INFO:teuthology.orchestra.run.smithi063.stdout:Get:19 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-markupsafe amd64 1.0-1build1 [13.5 kB] 2021-03-24T20:42:38.736 INFO:teuthology.orchestra.run.smithi052.stdout:Get:88 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-03-24T20:42:38.736 INFO:teuthology.orchestra.run.smithi052.stdout:Get:89 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-03-24T20:42:38.737 INFO:teuthology.orchestra.run.smithi052.stdout:Get:90 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-03-24T20:42:38.791 INFO:teuthology.orchestra.run.smithi063.stdout:Get:20 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-mako all 1.0.7+ds1-1 [59.3 kB] 2021-03-24T20:42:38.793 INFO:teuthology.orchestra.run.smithi063.stdout:Get:21 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-simplegeneric all 0.8.1-1 [11.5 kB] 2021-03-24T20:42:38.794 INFO:teuthology.orchestra.run.smithi063.stdout:Get:22 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-singledispatch all 3.4.0.3-2 [7,022 B] 2021-03-24T20:42:38.795 INFO:teuthology.orchestra.run.smithi063.stdout:Get:23 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webob all 1:1.7.3-2fakesync1 [64.3 kB] 2021-03-24T20:42:38.796 INFO:teuthology.orchestra.run.smithi063.stdout:Get:24 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-bs4 all 4.6.0-1 [67.8 kB] 2021-03-24T20:42:38.800 INFO:teuthology.orchestra.run.smithi063.stdout:Get:25 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-waitress all 1.0.1-1 [53.4 kB] 2021-03-24T20:42:38.870 INFO:teuthology.orchestra.run.smithi063.stdout:Get:26 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tempita all 0.5.2-2 [13.9 kB] 2021-03-24T20:42:38.871 INFO:teuthology.orchestra.run.smithi063.stdout:Get:27 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-paste all 2.0.3+dfsg-4ubuntu1 [456 kB] 2021-03-24T20:42:38.876 INFO:teuthology.orchestra.run.smithi052.stdout:Get:91 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-03-24T20:42:38.876 INFO:teuthology.orchestra.run.smithi052.stdout:Get:92 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-03-24T20:42:38.877 INFO:teuthology.orchestra.run.smithi052.stdout:Get:93 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.5 [330 kB] 2021-03-24T20:42:38.881 INFO:teuthology.orchestra.run.smithi052.stdout:Get:94 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-03-24T20:42:38.882 INFO:teuthology.orchestra.run.smithi052.stdout:Get:95 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-03-24T20:42:38.882 INFO:teuthology.orchestra.run.smithi052.stdout:Get:96 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-03-24T20:42:38.883 INFO:teuthology.orchestra.run.smithi052.stdout:Get:97 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-03-24T20:42:38.884 INFO:teuthology.orchestra.run.smithi052.stdout:Get:98 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-03-24T20:42:38.885 INFO:teuthology.orchestra.run.smithi052.stdout:Get:99 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-03-24T20:42:38.892 INFO:teuthology.orchestra.run.smithi063.stdout:Get:28 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-pastedeploy-tpl all 1.5.2-4 [4,796 B] 2021-03-24T20:42:38.892 INFO:teuthology.orchestra.run.smithi063.stdout:Get:29 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastedeploy all 1.5.2-4 [13.4 kB] 2021-03-24T20:42:38.950 INFO:teuthology.orchestra.run.smithi063.stdout:Get:30 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webtest all 2.0.28-1ubuntu1 [27.9 kB] 2021-03-24T20:42:38.951 INFO:teuthology.orchestra.run.smithi063.stdout:Get:31 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pecan all 1.2.1-2 [86.1 kB] 2021-03-24T20:42:38.954 INFO:teuthology.orchestra.run.smithi063.stdout:Get:32 http://archive.ubuntu.com/ubuntu bionic-updates/universe amd64 python3-werkzeug all 0.14.1+dfsg1-1ubuntu0.1 [174 kB] 2021-03-24T20:42:38.963 INFO:teuthology.orchestra.run.smithi063.stdout:Get:33 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-jinja2 all 2.10-1ubuntu0.18.04.1 [95.4 kB] 2021-03-24T20:42:38.967 INFO:teuthology.orchestra.run.smithi063.stdout:Get:34 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-repoze.lru all 0.7-1 [11.7 kB] 2021-03-24T20:42:38.969 INFO:teuthology.orchestra.run.smithi063.stdout:Get:35 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-routes all 2.4.1-1 [28.4 kB] 2021-03-24T20:42:39.031 INFO:teuthology.orchestra.run.smithi063.stdout:Get:36 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-numpy amd64 1:1.13.3-2ubuntu1 [1,943 kB] 2021-03-24T20:42:39.068 INFO:teuthology.orchestra.run.smithi052.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.2.10-76-gdcf77bd9-1bionic [3,948 B] 2021-03-24T20:42:39.069 INFO:teuthology.orchestra.run.smithi052.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.2.10-76-gdcf77bd9-1bionic [517 kB] 2021-03-24T20:42:39.081 INFO:teuthology.orchestra.run.smithi052.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.2.10-76-gdcf77bd9-1bionic [1,854 kB] 2021-03-24T20:42:39.127 INFO:teuthology.orchestra.run.smithi063.stdout:Get:37 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-decorator all 4.1.2-1 [9,364 B] 2021-03-24T20:42:39.128 INFO:teuthology.orchestra.run.smithi063.stdout:Get:38 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-scipy amd64 0.19.1-2ubuntu1 [9,619 kB] 2021-03-24T20:42:39.142 INFO:teuthology.orchestra.run.smithi052.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 15.2.10-76-gdcf77bd9-1bionic [52.3 kB] 2021-03-24T20:42:39.143 INFO:teuthology.orchestra.run.smithi052.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 15.2.10-76-gdcf77bd9-1bionic [58.0 kB] 2021-03-24T20:42:39.145 INFO:teuthology.orchestra.run.smithi052.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.2.10-76-gdcf77bd9-1bionic [3,197 kB] 2021-03-24T20:42:39.252 INFO:teuthology.orchestra.run.smithi052.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.2.10-76-gdcf77bd9-1bionic [34.3 kB] 2021-03-24T20:42:39.253 INFO:teuthology.orchestra.run.smithi052.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.2.10-76-gdcf77bd9-1bionic [7,885 kB] 2021-03-24T20:42:39.405 INFO:teuthology.orchestra.run.smithi063.stdout:Get:39 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn-lib amd64 0.19.1-3 [1,429 kB] 2021-03-24T20:42:39.467 INFO:teuthology.orchestra.run.smithi063.stdout:Get:40 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-joblib all 0.11-1 [109 kB] 2021-03-24T20:42:39.469 INFO:teuthology.orchestra.run.smithi063.stdout:Get:41 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-sklearn all 0.19.1-3 [1,373 kB] 2021-03-24T20:42:39.483 INFO:teuthology.orchestra.run.smithi063.stdout:Get:42 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-monotonic all 1.1-2 [5,332 B] 2021-03-24T20:42:39.484 INFO:teuthology.orchestra.run.smithi063.stdout:Get:43 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-fasteners all 0.12.0-3 [13.5 kB] 2021-03-24T20:42:39.484 INFO:teuthology.orchestra.run.smithi063.stdout:Get:44 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-httplib2 all 0.9.2+dfsg-1ubuntu0.3 [28.8 kB] 2021-03-24T20:42:39.485 INFO:teuthology.orchestra.run.smithi063.stdout:Get:45 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1 all 0.4.2-3 [46.8 kB] 2021-03-24T20:42:39.485 INFO:teuthology.orchestra.run.smithi063.stdout:Get:46 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyasn1-modules all 0.2.1-0.2 [32.9 kB] 2021-03-24T20:42:39.486 INFO:teuthology.orchestra.run.smithi063.stdout:Get:47 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-rsa all 3.4.2-1 [29.0 kB] 2021-03-24T20:42:39.486 INFO:teuthology.orchestra.run.smithi063.stdout:Get:48 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-sqlalchemy all 1.1.11+ds1-1ubuntu1 [680 kB] 2021-03-24T20:42:39.517 INFO:teuthology.orchestra.run.smithi052.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.2.10-76-gdcf77bd9-1bionic [4,064 B] 2021-03-24T20:42:39.518 INFO:teuthology.orchestra.run.smithi052.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.2.10-76-gdcf77bd9-1bionic [22.5 kB] 2021-03-24T20:42:39.518 INFO:teuthology.orchestra.run.smithi052.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.2.10-76-gdcf77bd9-1bionic [44.1 MB] 2021-03-24T20:42:39.565 INFO:teuthology.orchestra.run.smithi063.stdout:Get:49 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-oauth2client all 4.1.2-3 [61.5 kB] 2021-03-24T20:42:39.567 INFO:teuthology.orchestra.run.smithi063.stdout:Get:50 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-setuptools all 39.0.1-2 [248 kB] 2021-03-24T20:42:39.573 INFO:teuthology.orchestra.run.smithi063.stdout:Get:51 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-websocket all 0.44.0-0ubuntu2 [30.7 kB] 2021-03-24T20:42:39.574 INFO:teuthology.orchestra.run.smithi063.stdout:Get:52 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-kubernetes all 2.0.0-2ubuntu1 [181 kB] 2021-03-24T20:42:39.577 INFO:teuthology.orchestra.run.smithi063.stdout:Get:53 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-json-pointer all 1.10-1 [7,478 B] 2021-03-24T20:42:39.578 INFO:teuthology.orchestra.run.smithi063.stdout:Get:54 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-jsonpatch all 1.19+really1.16-1fakesync1 [13.4 kB] 2021-03-24T20:42:39.645 INFO:teuthology.orchestra.run.smithi063.stdout:Get:55 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libonig4 amd64 6.7.0-1 [119 kB] 2021-03-24T20:42:39.648 INFO:teuthology.orchestra.run.smithi063.stdout:Get:56 http://archive.ubuntu.com/ubuntu bionic/universe amd64 libjq1 amd64 1.5+dfsg-2 [111 kB] 2021-03-24T20:42:39.651 INFO:teuthology.orchestra.run.smithi063.stdout:Get:57 http://archive.ubuntu.com/ubuntu bionic/universe amd64 jq amd64 1.5+dfsg-2 [45.6 kB] 2021-03-24T20:42:39.652 INFO:teuthology.orchestra.run.smithi063.stdout:Get:58 http://archive.ubuntu.com/ubuntu bionic/main amd64 socat amd64 1.7.3.2-2ubuntu2 [342 kB] 2021-03-24T20:42:39.725 INFO:teuthology.orchestra.run.smithi063.stdout:Get:59 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 libxslt1.1 amd64 1.1.29-5ubuntu0.2 [150 kB] 2021-03-24T20:42:39.730 INFO:teuthology.orchestra.run.smithi063.stdout:Get:60 http://archive.ubuntu.com/ubuntu bionic/universe amd64 xmlstarlet amd64 1.6.1-2 [382 kB] 2021-03-24T20:42:39.738 INFO:teuthology.orchestra.run.smithi063.stdout:Get:61 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 nvme-cli amd64 1.5-1ubuntu1.1 [184 kB] 2021-03-24T20:42:39.741 INFO:teuthology.orchestra.run.smithi063.stdout:Get:62 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-attr all 17.4.0-2 [23.7 kB] 2021-03-24T20:42:39.741 INFO:teuthology.orchestra.run.smithi063.stdout:Get:63 http://archive.ubuntu.com/ubuntu bionic/main amd64 python-funcsigs all 1.0.2-4 [13.5 kB] 2021-03-24T20:42:39.742 INFO:teuthology.orchestra.run.smithi063.stdout:Get:64 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pluggy all 0.6.0-1 [12.2 kB] 2021-03-24T20:42:39.805 INFO:teuthology.orchestra.run.smithi063.stdout:Get:65 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-py all 1.5.2-1 [65.1 kB] 2021-03-24T20:42:39.807 INFO:teuthology.orchestra.run.smithi063.stdout:Get:66 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python-pytest all 3.3.2-2 [142 kB] 2021-03-24T20:42:39.810 INFO:teuthology.orchestra.run.smithi063.stdout:Get:67 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-attr all 17.4.0-2 [23.8 kB] 2021-03-24T20:42:39.811 INFO:teuthology.orchestra.run.smithi063.stdout:Get:68 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-cycler all 0.10.0-1 [7,622 B] 2021-03-24T20:42:39.885 INFO:teuthology.orchestra.run.smithi063.stdout:Get:69 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-webencodings all 0.5-2 [10.4 kB] 2021-03-24T20:42:39.886 INFO:teuthology.orchestra.run.smithi063.stdout:Get:70 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-html5lib all 0.999999999-1 [81.9 kB] 2021-03-24T20:42:39.888 INFO:teuthology.orchestra.run.smithi063.stdout:Get:71 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-lxml amd64 4.2.1-1ubuntu0.3 [1,097 kB] 2021-03-24T20:42:39.908 INFO:teuthology.orchestra.run.smithi063.stdout:Get:72 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyparsing all 2.2.0+dfsg1-2 [52.2 kB] 2021-03-24T20:42:39.909 INFO:teuthology.orchestra.run.smithi063.stdout:Get:73 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-tz all 2018.3-2 [25.1 kB] 2021-03-24T20:42:39.910 INFO:teuthology.orchestra.run.smithi063.stdout:Get:74 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-matplotlib amd64 2.1.1-2ubuntu3 [3,907 kB] 2021-03-24T20:42:40.054 INFO:teuthology.orchestra.run.smithi063.stdout:Get:75 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-olefile all 0.45.1-1 [33.3 kB] 2021-03-24T20:42:40.055 INFO:teuthology.orchestra.run.smithi063.stdout:Get:76 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pastescript all 2.0.2-2 [54.6 kB] 2021-03-24T20:42:40.056 INFO:teuthology.orchestra.run.smithi063.stdout:Get:77 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-pil amd64 5.1.0-1ubuntu0.5 [330 kB] 2021-03-24T20:42:40.061 INFO:teuthology.orchestra.run.smithi063.stdout:Get:78 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pluggy all 0.6.0-1 [12.3 kB] 2021-03-24T20:42:40.061 INFO:teuthology.orchestra.run.smithi063.stdout:Get:79 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-py all 1.5.2-1 [65.2 kB] 2021-03-24T20:42:40.062 INFO:teuthology.orchestra.run.smithi063.stdout:Get:80 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-pyinotify all 0.9.6-1 [24.7 kB] 2021-03-24T20:42:40.062 INFO:teuthology.orchestra.run.smithi063.stdout:Get:81 http://archive.ubuntu.com/ubuntu bionic/universe amd64 python3-pytest all 3.3.2-2 [142 kB] 2021-03-24T20:42:40.063 INFO:teuthology.orchestra.run.smithi063.stdout:Get:82 http://archive.ubuntu.com/ubuntu bionic/main amd64 python3-simplejson amd64 3.13.2-1 [49.6 kB] 2021-03-24T20:42:40.064 INFO:teuthology.orchestra.run.smithi063.stdout:Get:83 http://archive.ubuntu.com/ubuntu bionic-updates/main amd64 python3-tk amd64 3.6.9-1~18.04 [119 kB] 2021-03-24T20:42:40.997 INFO:teuthology.orchestra.run.smithi052.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.2.10-76-gdcf77bd9-1bionic [18.0 kB] 2021-03-24T20:42:40.998 INFO:teuthology.orchestra.run.smithi052.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.2.10-76-gdcf77bd9-1bionic [8,883 kB] 2021-03-24T20:42:41.290 INFO:teuthology.orchestra.run.smithi052.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.2.10-76-gdcf77bd9-1bionic [63.1 kB] 2021-03-24T20:42:41.372 INFO:teuthology.orchestra.run.smithi063.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 librados2 amd64 15.2.10-76-gdcf77bd9-1bionic [3,060 kB] 2021-03-24T20:42:42.024 INFO:teuthology.orchestra.run.smithi052.stdout:Fetched 156 MB in 5s (34.3 MB/s) 2021-03-24T20:42:42.171 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-uritemplate. 2021-03-24T20:42:42.221 INFO:teuthology.orchestra.run.smithi052.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137597 files and directories currently installed.) 2021-03-24T20:42:42.225 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-03-24T20:42:42.269 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-03-24T20:42:42.275 INFO:teuthology.orchestra.run.smithi063.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs2 amd64 15.2.10-76-gdcf77bd9-1bionic [435 kB] 2021-03-24T20:42:42.376 INFO:teuthology.orchestra.run.smithi063.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 python3-rados amd64 15.2.10-76-gdcf77bd9-1bionic [345 kB] 2021-03-24T20:42:42.455 INFO:teuthology.orchestra.run.smithi063.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-argparse all 15.2.10-76-gdcf77bd9-1bionic [20.7 kB] 2021-03-24T20:42:42.456 INFO:teuthology.orchestra.run.smithi063.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 python3-cephfs amd64 15.2.10-76-gdcf77bd9-1bionic [133 kB] 2021-03-24T20:42:42.493 INFO:teuthology.orchestra.run.smithi063.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 python3-ceph-common all 15.2.10-76-gdcf77bd9-1bionic [29.1 kB] 2021-03-24T20:42:42.495 INFO:teuthology.orchestra.run.smithi063.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 python3-rbd amd64 15.2.10-76-gdcf77bd9-1bionic [326 kB] 2021-03-24T20:42:42.556 INFO:teuthology.orchestra.run.smithi063.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 librgw2 amd64 15.2.10-76-gdcf77bd9-1bionic [2,902 kB] 2021-03-24T20:42:42.622 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-03-24T20:42:42.636 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-03-24T20:42:42.723 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-24T20:42:43.042 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../002-librbd1_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:43.116 INFO:teuthology.orchestra.run.smithi063.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 python3-rgw amd64 15.2.10-76-gdcf77bd9-1bionic [99.4 kB] 2021-03-24T20:42:43.136 INFO:teuthology.orchestra.run.smithi063.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 libradosstriper1 amd64 15.2.10-76-gdcf77bd9-1bionic [338 kB] 2021-03-24T20:42:43.167 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking librbd1 (15.2.10-76-gdcf77bd9-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-24T20:42:43.195 INFO:teuthology.orchestra.run.smithi063.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-common amd64 15.2.10-76-gdcf77bd9-1bionic [18.8 MB] 2021-03-24T20:42:43.710 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../003-librados2_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:43.838 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking librados2 (15.2.10-76-gdcf77bd9-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-24T20:42:44.483 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libcephfs2. 2021-03-24T20:42:44.497 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../004-libcephfs2_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:44.542 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libcephfs2 (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:44.827 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-rados. 2021-03-24T20:42:44.841 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../005-python3-rados_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:44.919 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-rados (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:45.280 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-03-24T20:42:45.293 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../006-python3-ceph-argparse_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:42:45.330 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-ceph-argparse (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:45.559 INFO:teuthology.orchestra.run.smithi063.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-base amd64 15.2.10-76-gdcf77bd9-1bionic [5,176 kB] 2021-03-24T20:42:45.632 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-cephfs. 2021-03-24T20:42:45.649 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../007-python3-cephfs_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:45.691 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-cephfs (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:46.018 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-ceph-common. 2021-03-24T20:42:46.033 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../008-python3-ceph-common_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:42:46.052 INFO:teuthology.orchestra.run.smithi063.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-modules-core all 15.2.10-76-gdcf77bd9-1bionic [161 kB] 2021-03-24T20:42:46.066 INFO:teuthology.orchestra.run.smithi063.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr amd64 15.2.10-76-gdcf77bd9-1bionic [1,284 kB] 2021-03-24T20:42:46.076 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-ceph-common (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:46.186 INFO:teuthology.orchestra.run.smithi063.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mon amd64 15.2.10-76-gdcf77bd9-1bionic [5,943 kB] 2021-03-24T20:42:46.388 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-prettytable. 2021-03-24T20:42:46.401 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2021-03-24T20:42:46.438 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-03-24T20:42:46.690 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-rbd. 2021-03-24T20:42:46.725 INFO:teuthology.orchestra.run.smithi063.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-osd amd64 15.2.10-76-gdcf77bd9-1bionic [22.8 MB] 2021-03-24T20:42:46.726 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../010-python3-rbd_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:46.748 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-rbd (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:47.118 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-03-24T20:42:47.200 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-03-24T20:42:47.201 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-24T20:42:47.529 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-03-24T20:42:47.544 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-03-24T20:42:47.597 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-24T20:42:47.940 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package librgw2. 2021-03-24T20:42:47.953 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../013-librgw2_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:47.990 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking librgw2 (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:48.400 INFO:teuthology.orchestra.run.smithi063.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph amd64 15.2.10-76-gdcf77bd9-1bionic [3,948 B] 2021-03-24T20:42:48.401 INFO:teuthology.orchestra.run.smithi063.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-fuse amd64 15.2.10-76-gdcf77bd9-1bionic [517 kB] 2021-03-24T20:42:48.434 INFO:teuthology.orchestra.run.smithi063.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mds amd64 15.2.10-76-gdcf77bd9-1bionic [1,854 kB] 2021-03-24T20:42:48.556 INFO:teuthology.orchestra.run.smithi063.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 cephadm amd64 15.2.10-76-gdcf77bd9-1bionic [52.3 kB] 2021-03-24T20:42:48.559 INFO:teuthology.orchestra.run.smithi063.stdout:Get:104 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-cephadm all 15.2.10-76-gdcf77bd9-1bionic [58.0 kB] 2021-03-24T20:42:48.561 INFO:teuthology.orchestra.run.smithi063.stdout:Get:105 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-dashboard all 15.2.10-76-gdcf77bd9-1bionic [3,197 kB] 2021-03-24T20:42:48.634 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-rgw. 2021-03-24T20:42:48.647 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../014-python3-rgw_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:48.684 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-rgw (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:48.772 INFO:teuthology.orchestra.run.smithi063.stdout:Get:106 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-cloud all 15.2.10-76-gdcf77bd9-1bionic [34.3 kB] 2021-03-24T20:42:48.773 INFO:teuthology.orchestra.run.smithi063.stdout:Get:107 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-diskprediction-local all 15.2.10-76-gdcf77bd9-1bionic [7,885 kB] 2021-03-24T20:42:49.019 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-03-24T20:42:49.033 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-03-24T20:42:49.070 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-24T20:42:49.273 INFO:teuthology.orchestra.run.smithi063.stdout:Get:108 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-k8sevents all 15.2.10-76-gdcf77bd9-1bionic [4,064 B] 2021-03-24T20:42:49.274 INFO:teuthology.orchestra.run.smithi063.stdout:Get:109 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-mgr-rook all 15.2.10-76-gdcf77bd9-1bionic [22.5 kB] 2021-03-24T20:42:49.274 INFO:teuthology.orchestra.run.smithi063.stdout:Get:110 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 ceph-test amd64 15.2.10-76-gdcf77bd9-1bionic [44.1 MB] 2021-03-24T20:42:49.380 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-03-24T20:42:49.395 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2021-03-24T20:42:49.439 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-03-24T20:42:49.842 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package liboath0. 2021-03-24T20:42:49.856 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2021-03-24T20:42:49.900 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-03-24T20:42:50.219 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libradosstriper1. 2021-03-24T20:42:50.233 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../018-libradosstriper1_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:50.285 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libradosstriper1 (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:50.722 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-common. 2021-03-24T20:42:50.738 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../019-ceph-common_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:50.780 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-common (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:51.370 INFO:teuthology.orchestra.run.smithi063.stdout:Get:111 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 libcephfs-dev amd64 15.2.10-76-gdcf77bd9-1bionic [18.0 kB] 2021-03-24T20:42:51.371 INFO:teuthology.orchestra.run.smithi063.stdout:Get:112 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 radosgw amd64 15.2.10-76-gdcf77bd9-1bionic [8,883 kB] 2021-03-24T20:42:51.671 INFO:teuthology.orchestra.run.smithi063.stdout:Get:113 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic/main amd64 rbd-fuse amd64 15.2.10-76-gdcf77bd9-1bionic [63.1 kB] 2021-03-24T20:42:52.381 INFO:teuthology.orchestra.run.smithi063.stdout:Fetched 156 MB in 14s (11.4 MB/s) 2021-03-24T20:42:52.531 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-uritemplate. 2021-03-24T20:42:52.536 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-base. 2021-03-24T20:42:52.549 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../020-ceph-base_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:52.578 INFO:teuthology.orchestra.run.smithi063.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 137597 files and directories currently installed.) 2021-03-24T20:42:52.582 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../000-python3-uritemplate_0.6-1ubuntu1_all.deb ... 2021-03-24T20:42:52.603 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-base (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:52.621 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-uritemplate (0.6-1ubuntu1) ... 2021-03-24T20:42:52.940 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2021-03-24T20:42:52.954 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../001-liblttng-ust0_2.10.1-1_amd64.deb ... 2021-03-24T20:42:53.040 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-24T20:42:53.391 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../002-librbd1_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:53.473 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-dateutil. 2021-03-24T20:42:53.488 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2021-03-24T20:42:53.531 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-03-24T20:42:53.661 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking librbd1 (15.2.10-76-gdcf77bd9-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-24T20:42:53.851 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-asn1crypto. 2021-03-24T20:42:53.865 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2021-03-24T20:42:53.909 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-03-24T20:42:54.203 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-cffi-backend. 2021-03-24T20:42:54.204 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../003-librados2_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:54.217 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-03-24T20:42:54.261 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-03-24T20:42:54.331 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking librados2 (15.2.10-76-gdcf77bd9-1bionic) over (12.2.13-0ubuntu0.18.04.6) ... 2021-03-24T20:42:54.589 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-cryptography. 2021-03-24T20:42:54.604 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-03-24T20:42:54.647 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-24T20:42:54.943 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libcephfs2. 2021-03-24T20:42:54.959 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../004-libcephfs2_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:55.001 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libcephfs2 (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:55.042 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-openssl. 2021-03-24T20:42:55.058 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-03-24T20:42:55.100 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-03-24T20:42:55.353 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-rados. 2021-03-24T20:42:55.367 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../005-python3-rados_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:55.386 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-03-24T20:42:55.402 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:42:55.420 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-rados (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:55.453 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-modules-core (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:55.764 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-ceph-argparse. 2021-03-24T20:42:55.778 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../006-python3-ceph-argparse_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:42:55.815 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-bcrypt. 2021-03-24T20:42:55.822 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-ceph-argparse (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:55.829 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-03-24T20:42:55.873 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-03-24T20:42:56.142 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-cephfs. 2021-03-24T20:42:56.155 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../007-python3-cephfs_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:56.184 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-cherrypy3. 2021-03-24T20:42:56.197 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2021-03-24T20:42:56.200 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-cephfs (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:56.242 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-03-24T20:42:56.536 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-ceph-common. 2021-03-24T20:42:56.550 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../008-python3-ceph-common_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:42:56.603 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-ceph-common (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:56.604 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-jwt. 2021-03-24T20:42:56.617 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-03-24T20:42:56.654 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-03-24T20:42:56.972 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-prettytable. 2021-03-24T20:42:56.973 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-logutils. 2021-03-24T20:42:56.986 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../009-python3-prettytable_0.7.2-3_all.deb ... 2021-03-24T20:42:56.989 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2021-03-24T20:42:57.032 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-03-24T20:42:57.038 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-prettytable (0.7.2-3) ... 2021-03-24T20:42:57.299 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-rbd. 2021-03-24T20:42:57.314 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../010-python3-rbd_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:57.342 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-markupsafe. 2021-03-24T20:42:57.356 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-03-24T20:42:57.357 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-rbd (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:57.401 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-03-24T20:42:57.703 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-mako. 2021-03-24T20:42:57.717 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2021-03-24T20:42:57.761 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-03-24T20:42:57.810 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package librabbitmq4:amd64. 2021-03-24T20:42:57.823 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../011-librabbitmq4_0.8.0-1ubuntu0.18.04.2_amd64.deb ... 2021-03-24T20:42:57.860 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-24T20:42:58.081 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-simplegeneric. 2021-03-24T20:42:58.096 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2021-03-24T20:42:58.139 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-03-24T20:42:58.188 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package librdkafka1:amd64. 2021-03-24T20:42:58.202 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../012-librdkafka1_0.11.3-1build1_amd64.deb ... 2021-03-24T20:42:58.246 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-24T20:42:58.442 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-singledispatch. 2021-03-24T20:42:58.456 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-03-24T20:42:58.500 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-03-24T20:42:58.649 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package librgw2. 2021-03-24T20:42:58.663 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../013-librgw2_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:58.707 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking librgw2 (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:58.819 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-webob. 2021-03-24T20:42:58.835 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-03-24T20:42:58.877 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-03-24T20:42:59.197 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-bs4. 2021-03-24T20:42:59.212 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2021-03-24T20:42:59.251 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-rgw. 2021-03-24T20:42:59.308 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-03-24T20:42:59.308 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../014-python3-rgw_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:42:59.309 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-rgw (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:42:59.541 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-waitress. 2021-03-24T20:42:59.554 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2021-03-24T20:42:59.582 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-03-24T20:42:59.637 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libsnappy1v5:amd64. 2021-03-24T20:42:59.651 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../015-libsnappy1v5_1.1.7-1_amd64.deb ... 2021-03-24T20:42:59.695 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-24T20:42:59.843 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-tempita. 2021-03-24T20:42:59.860 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2021-03-24T20:42:59.893 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-03-24T20:42:59.997 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libleveldb1v5:amd64. 2021-03-24T20:43:00.011 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../016-libleveldb1v5_1.20-2_amd64.deb ... 2021-03-24T20:43:00.072 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libleveldb1v5:amd64 (1.20-2) ... 2021-03-24T20:43:00.154 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-paste. 2021-03-24T20:43:00.169 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-03-24T20:43:00.212 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-24T20:43:00.475 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package liboath0. 2021-03-24T20:43:00.490 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../017-liboath0_2.6.1-1_amd64.deb ... 2021-03-24T20:43:00.533 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking liboath0 (2.6.1-1) ... 2021-03-24T20:43:00.541 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-03-24T20:43:00.557 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-03-24T20:43:00.590 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-03-24T20:43:00.827 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libradosstriper1. 2021-03-24T20:43:00.840 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../018-libradosstriper1_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:00.886 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libradosstriper1 (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:00.918 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pastedeploy. 2021-03-24T20:43:00.932 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2021-03-24T20:43:00.984 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-03-24T20:43:01.320 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-webtest. 2021-03-24T20:43:01.330 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-common. 2021-03-24T20:43:01.337 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-03-24T20:43:01.346 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../019-ceph-common_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:01.379 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-03-24T20:43:01.389 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-common (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:01.723 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pecan. 2021-03-24T20:43:01.737 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2021-03-24T20:43:01.781 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-03-24T20:43:02.143 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-werkzeug. 2021-03-24T20:43:02.157 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-03-24T20:43:02.201 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-24T20:43:02.554 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr. 2021-03-24T20:43:02.570 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../045-ceph-mgr_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:02.612 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:03.165 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mon. 2021-03-24T20:43:03.180 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../046-ceph-mon_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:03.223 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mon (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:03.369 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-base. 2021-03-24T20:43:03.382 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../020-ceph-base_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:03.435 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-base (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:04.323 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-dateutil. 2021-03-24T20:43:04.338 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../021-python3-dateutil_2.6.1-1_all.deb ... 2021-03-24T20:43:04.381 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-dateutil (2.6.1-1) ... 2021-03-24T20:43:04.453 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-osd. 2021-03-24T20:43:04.465 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../047-ceph-osd_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:04.518 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-osd (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:04.709 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-asn1crypto. 2021-03-24T20:43:04.725 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../022-python3-asn1crypto_0.24.0-1_all.deb ... 2021-03-24T20:43:04.784 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-asn1crypto (0.24.0-1) ... 2021-03-24T20:43:05.062 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-cffi-backend. 2021-03-24T20:43:05.078 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../023-python3-cffi-backend_1.11.5-1_amd64.deb ... 2021-03-24T20:43:05.120 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-cffi-backend (1.11.5-1) ... 2021-03-24T20:43:05.481 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-cryptography. 2021-03-24T20:43:05.495 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../024-python3-cryptography_2.1.4-1ubuntu1.4_amd64.deb ... 2021-03-24T20:43:05.547 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-24T20:43:05.959 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-openssl. 2021-03-24T20:43:05.972 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../025-python3-openssl_17.5.0-1ubuntu1_all.deb ... 2021-03-24T20:43:06.017 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-openssl (17.5.0-1ubuntu1) ... 2021-03-24T20:43:06.277 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2021-03-24T20:43:06.291 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../026-ceph-mgr-modules-core_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:43:06.369 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr-modules-core (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:06.454 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph. 2021-03-24T20:43:06.469 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../048-ceph_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:06.512 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:06.773 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-bcrypt. 2021-03-24T20:43:06.786 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../027-python3-bcrypt_3.1.4-2_amd64.deb ... 2021-03-24T20:43:06.823 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-fuse. 2021-03-24T20:43:06.831 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-bcrypt (3.1.4-2) ... 2021-03-24T20:43:06.839 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../049-ceph-fuse_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:06.881 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-fuse (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:07.150 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-cherrypy3. 2021-03-24T20:43:07.165 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../028-python3-cherrypy3_8.9.1-2_all.deb ... 2021-03-24T20:43:07.209 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-cherrypy3 (8.9.1-2) ... 2021-03-24T20:43:07.268 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mds. 2021-03-24T20:43:07.283 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../050-ceph-mds_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:07.326 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mds (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:07.562 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-jwt. 2021-03-24T20:43:07.577 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../029-python3-jwt_1.5.3+ds1-1_all.deb ... 2021-03-24T20:43:07.620 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-jwt (1.5.3+ds1-1) ... 2021-03-24T20:43:07.871 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package cephadm. 2021-03-24T20:43:07.885 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../051-cephadm_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:07.954 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking cephadm (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:07.956 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-logutils. 2021-03-24T20:43:07.971 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../030-python3-logutils_0.3.3-5_all.deb ... 2021-03-24T20:43:08.015 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-logutils (0.3.3-5) ... 2021-03-24T20:43:08.257 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-jinja2. 2021-03-24T20:43:08.273 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-03-24T20:43:08.315 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-24T20:43:08.317 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-markupsafe. 2021-03-24T20:43:08.333 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../031-python3-markupsafe_1.0-1build1_amd64.deb ... 2021-03-24T20:43:08.618 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-03-24T20:43:08.634 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:43:08.676 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-cephadm (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:09.005 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-repoze.lru. 2021-03-24T20:43:09.018 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2021-03-24T20:43:09.025 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-markupsafe (1.0-1build1) ... 2021-03-24T20:43:09.070 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-03-24T20:43:09.312 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-mako. 2021-03-24T20:43:09.328 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../032-python3-mako_1.0.7+ds1-1_all.deb ... 2021-03-24T20:43:09.378 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-mako (1.0.7+ds1-1) ... 2021-03-24T20:43:09.756 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-simplegeneric. 2021-03-24T20:43:09.774 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../033-python3-simplegeneric_0.8.1-1_all.deb ... 2021-03-24T20:43:09.822 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-simplegeneric (0.8.1-1) ... 2021-03-24T20:43:10.048 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-routes. 2021-03-24T20:43:10.061 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2021-03-24T20:43:10.106 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-03-24T20:43:10.125 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-singledispatch. 2021-03-24T20:43:10.139 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../034-python3-singledispatch_3.4.0.3-2_all.deb ... 2021-03-24T20:43:10.183 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-singledispatch (3.4.0.3-2) ... 2021-03-24T20:43:10.453 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-03-24T20:43:10.466 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:43:10.477 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-webob. 2021-03-24T20:43:10.491 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../035-python3-webob_1%3a1.7.3-2fakesync1_all.deb ... 2021-03-24T20:43:10.509 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-dashboard (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:10.535 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-webob (1:1.7.3-2fakesync1) ... 2021-03-24T20:43:10.846 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-bs4. 2021-03-24T20:43:10.861 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../036-python3-bs4_4.6.0-1_all.deb ... 2021-03-24T20:43:10.905 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-bs4 (4.6.0-1) ... 2021-03-24T20:43:11.257 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-waitress. 2021-03-24T20:43:11.271 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../037-python3-waitress_1.0.1-1_all.deb ... 2021-03-24T20:43:11.316 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-waitress (1.0.1-1) ... 2021-03-24T20:43:11.426 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2021-03-24T20:43:11.441 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../057-ceph-mgr-diskprediction-cloud_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:43:11.493 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:11.627 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-tempita. 2021-03-24T20:43:11.643 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../038-python3-tempita_0.5.2-2_all.deb ... 2021-03-24T20:43:11.685 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-tempita (0.5.2-2) ... 2021-03-24T20:43:11.804 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-numpy. 2021-03-24T20:43:11.819 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-03-24T20:43:11.862 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-24T20:43:12.021 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-paste. 2021-03-24T20:43:12.037 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../039-python3-paste_2.0.3+dfsg-4ubuntu1_all.deb ... 2021-03-24T20:43:12.079 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-24T20:43:12.399 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2021-03-24T20:43:12.414 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../040-python-pastedeploy-tpl_1.5.2-4_all.deb ... 2021-03-24T20:43:12.457 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python-pastedeploy-tpl (1.5.2-4) ... 2021-03-24T20:43:12.477 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-decorator. 2021-03-24T20:43:12.493 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-03-24T20:43:12.560 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-03-24T20:43:12.809 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pastedeploy. 2021-03-24T20:43:12.823 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../041-python3-pastedeploy_1.5.2-4_all.deb ... 2021-03-24T20:43:12.876 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pastedeploy (1.5.2-4) ... 2021-03-24T20:43:12.889 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-scipy. 2021-03-24T20:43:12.901 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-03-24T20:43:12.946 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-03-24T20:43:13.187 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-webtest. 2021-03-24T20:43:13.202 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../042-python3-webtest_2.0.28-1ubuntu1_all.deb ... 2021-03-24T20:43:13.245 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-webtest (2.0.28-1ubuntu1) ... 2021-03-24T20:43:13.581 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pecan. 2021-03-24T20:43:13.595 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../043-python3-pecan_1.2.1-2_all.deb ... 2021-03-24T20:43:13.639 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pecan (1.2.1-2) ... 2021-03-24T20:43:13.992 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-werkzeug. 2021-03-24T20:43:14.007 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../044-python3-werkzeug_0.14.1+dfsg1-1ubuntu0.1_all.deb ... 2021-03-24T20:43:14.051 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-24T20:43:14.255 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-03-24T20:43:14.271 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-03-24T20:43:14.372 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-03-24T20:43:14.429 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr. 2021-03-24T20:43:14.443 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../045-ceph-mgr_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:14.487 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:14.858 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-joblib. 2021-03-24T20:43:14.873 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-03-24T20:43:14.916 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-joblib (0.11-1) ... 2021-03-24T20:43:15.065 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mon. 2021-03-24T20:43:15.079 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../046-ceph-mon_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:15.123 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mon (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:15.286 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-sklearn. 2021-03-24T20:43:15.300 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-03-24T20:43:15.345 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-03-24T20:43:15.876 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-03-24T20:43:15.894 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:43:15.934 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:16.193 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-osd. 2021-03-24T20:43:16.207 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../047-ceph-osd_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:16.251 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-osd (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:17.070 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-monotonic. 2021-03-24T20:43:17.086 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-03-24T20:43:17.153 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-03-24T20:43:17.456 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-fasteners. 2021-03-24T20:43:17.472 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-03-24T20:43:17.523 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-03-24T20:43:17.842 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-httplib2. 2021-03-24T20:43:17.857 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb ... 2021-03-24T20:43:17.900 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-24T20:43:18.141 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph. 2021-03-24T20:43:18.154 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../048-ceph_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:18.199 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:18.203 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pyasn1. 2021-03-24T20:43:18.217 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-03-24T20:43:18.261 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-03-24T20:43:18.518 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-fuse. 2021-03-24T20:43:18.531 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../049-ceph-fuse_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:18.590 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-03-24T20:43:18.606 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-03-24T20:43:18.648 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-24T20:43:18.777 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-fuse (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:18.959 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-rsa. 2021-03-24T20:43:18.975 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-03-24T20:43:19.018 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-03-24T20:43:19.197 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mds. 2021-03-24T20:43:19.211 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../050-ceph-mds_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:19.255 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mds (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:19.329 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-03-24T20:43:19.344 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-03-24T20:43:19.387 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-24T20:43:19.799 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package cephadm. 2021-03-24T20:43:19.816 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../051-cephadm_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:19.858 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking cephadm (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:19.858 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-oauth2client. 2021-03-24T20:43:19.874 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-03-24T20:43:19.916 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-03-24T20:43:20.185 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-jinja2. 2021-03-24T20:43:20.200 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../052-python3-jinja2_2.10-1ubuntu0.18.04.1_all.deb ... 2021-03-24T20:43:20.245 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-setuptools. 2021-03-24T20:43:20.298 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-24T20:43:20.298 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-03-24T20:43:20.302 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-03-24T20:43:20.605 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2021-03-24T20:43:20.620 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../053-ceph-mgr-cephadm_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:43:20.672 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr-cephadm (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:20.689 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-websocket. 2021-03-24T20:43:20.706 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-03-24T20:43:20.748 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-03-24T20:43:21.033 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-repoze.lru. 2021-03-24T20:43:21.047 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../054-python3-repoze.lru_0.7-1_all.deb ... 2021-03-24T20:43:21.084 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-kubernetes. 2021-03-24T20:43:21.091 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-repoze.lru (0.7-1) ... 2021-03-24T20:43:21.100 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-03-24T20:43:21.151 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-24T20:43:21.397 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-routes. 2021-03-24T20:43:21.413 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../055-python3-routes_2.4.1-1_all.deb ... 2021-03-24T20:43:21.456 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-routes (2.4.1-1) ... 2021-03-24T20:43:21.571 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-03-24T20:43:21.585 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:43:21.638 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-k8sevents (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:21.783 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2021-03-24T20:43:21.798 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../056-ceph-mgr-dashboard_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:43:21.850 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr-dashboard (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:21.974 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-json-pointer. 2021-03-24T20:43:21.989 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-03-24T20:43:22.032 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-03-24T20:43:22.385 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-jsonpatch. 2021-03-24T20:43:22.401 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-03-24T20:43:22.447 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-24T20:43:22.747 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-03-24T20:43:22.755 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2021-03-24T20:43:22.761 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../079-ceph-mgr-rook_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:43:22.772 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../057-ceph-mgr-diskprediction-cloud_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:43:22.813 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-mgr-rook (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:22.814 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:23.116 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-numpy. 2021-03-24T20:43:23.130 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../058-python3-numpy_1%3a1.13.3-2ubuntu1_amd64.deb ... 2021-03-24T20:43:23.151 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libonig4:amd64. 2021-03-24T20:43:23.168 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-03-24T20:43:23.174 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-24T20:43:23.286 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-03-24T20:43:23.623 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libjq1:amd64. 2021-03-24T20:43:23.639 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-03-24T20:43:23.681 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-03-24T20:43:23.894 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-decorator. 2021-03-24T20:43:23.911 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../059-python3-decorator_4.1.2-1_all.deb ... 2021-03-24T20:43:23.955 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-decorator (4.1.2-1) ... 2021-03-24T20:43:23.967 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package jq. 2021-03-24T20:43:23.982 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-03-24T20:43:24.034 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-03-24T20:43:24.241 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-scipy. 2021-03-24T20:43:24.255 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../060-python3-scipy_0.19.1-2ubuntu1_amd64.deb ... 2021-03-24T20:43:24.286 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package socat. 2021-03-24T20:43:24.291 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-scipy (0.19.1-2ubuntu1) ... 2021-03-24T20:43:24.301 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-03-24T20:43:24.344 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-03-24T20:43:24.739 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-03-24T20:43:24.753 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-03-24T20:43:24.797 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-24T20:43:25.050 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package xmlstarlet. 2021-03-24T20:43:25.066 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-03-24T20:43:25.117 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-03-24T20:43:25.446 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package ceph-test. 2021-03-24T20:43:25.461 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../086-ceph-test_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:25.495 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-sklearn-lib. 2021-03-24T20:43:25.505 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking ceph-test (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:25.506 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../061-python3-sklearn-lib_0.19.1-3_amd64.deb ... 2021-03-24T20:43:25.536 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-sklearn-lib (0.19.1-3) ... 2021-03-24T20:43:25.932 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-joblib. 2021-03-24T20:43:25.945 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../062-python3-joblib_0.11-1_all.deb ... 2021-03-24T20:43:25.990 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-joblib (0.11-1) ... 2021-03-24T20:43:26.343 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-sklearn. 2021-03-24T20:43:26.360 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../063-python3-sklearn_0.19.1-3_all.deb ... 2021-03-24T20:43:26.401 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-sklearn (0.19.1-3) ... 2021-03-24T20:43:26.932 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2021-03-24T20:43:26.949 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../064-ceph-mgr-diskprediction-local_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:43:26.991 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:28.135 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-monotonic. 2021-03-24T20:43:28.151 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../065-python3-monotonic_1.1-2_all.deb ... 2021-03-24T20:43:28.185 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-monotonic (1.1-2) ... 2021-03-24T20:43:28.491 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-fasteners. 2021-03-24T20:43:28.504 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../066-python3-fasteners_0.12.0-3_all.deb ... 2021-03-24T20:43:28.557 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-fasteners (0.12.0-3) ... 2021-03-24T20:43:28.893 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-httplib2. 2021-03-24T20:43:28.907 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../067-python3-httplib2_0.9.2+dfsg-1ubuntu0.3_all.deb ... 2021-03-24T20:43:28.908 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package libcephfs-dev. 2021-03-24T20:43:28.925 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../087-libcephfs-dev_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:28.966 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking libcephfs-dev (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:28.993 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-24T20:43:29.235 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package nvme-cli. 2021-03-24T20:43:29.253 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-03-24T20:43:29.279 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pyasn1. 2021-03-24T20:43:29.293 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-03-24T20:43:29.296 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../068-python3-pyasn1_0.4.2-3_all.deb ... 2021-03-24T20:43:29.337 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pyasn1 (0.4.2-3) ... 2021-03-24T20:43:29.664 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-attr. 2021-03-24T20:43:29.674 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pyasn1-modules. 2021-03-24T20:43:29.680 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-03-24T20:43:29.691 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../069-python3-pyasn1-modules_0.2.1-0.2_all.deb ... 2021-03-24T20:43:29.723 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-attr (17.4.0-2) ... 2021-03-24T20:43:29.732 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-24T20:43:30.034 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-funcsigs. 2021-03-24T20:43:30.043 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-rsa. 2021-03-24T20:43:30.048 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-03-24T20:43:30.059 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../070-python3-rsa_3.4.2-1_all.deb ... 2021-03-24T20:43:30.092 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-03-24T20:43:30.101 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-rsa (3.4.2-1) ... 2021-03-24T20:43:30.403 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-pluggy. 2021-03-24T20:43:30.419 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-03-24T20:43:30.438 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-sqlalchemy. 2021-03-24T20:43:30.451 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../071-python3-sqlalchemy_1.1.11+ds1-1ubuntu1_all.deb ... 2021-03-24T20:43:30.462 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-03-24T20:43:30.496 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-24T20:43:30.756 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-py. 2021-03-24T20:43:30.771 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-03-24T20:43:30.814 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-py (1.5.2-1) ... 2021-03-24T20:43:30.950 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-oauth2client. 2021-03-24T20:43:30.964 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../072-python3-oauth2client_4.1.2-3_all.deb ... 2021-03-24T20:43:31.008 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-oauth2client (4.1.2-3) ... 2021-03-24T20:43:31.168 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python-pytest. 2021-03-24T20:43:31.182 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-03-24T20:43:31.226 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-03-24T20:43:31.353 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-setuptools. 2021-03-24T20:43:31.368 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../073-python3-setuptools_39.0.1-2_all.deb ... 2021-03-24T20:43:31.411 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-setuptools (39.0.1-2) ... 2021-03-24T20:43:31.564 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-attr. 2021-03-24T20:43:31.577 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-03-24T20:43:31.622 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-03-24T20:43:31.831 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-websocket. 2021-03-24T20:43:31.848 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../074-python3-websocket_0.44.0-0ubuntu2_all.deb ... 2021-03-24T20:43:31.898 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-websocket (0.44.0-0ubuntu2) ... 2021-03-24T20:43:31.948 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-cycler. 2021-03-24T20:43:31.964 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-03-24T20:43:32.015 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-03-24T20:43:32.209 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-kubernetes. 2021-03-24T20:43:32.223 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../075-python3-kubernetes_2.0.0-2ubuntu1_all.deb ... 2021-03-24T20:43:32.267 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-24T20:43:32.326 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-webencodings. 2021-03-24T20:43:32.341 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-03-24T20:43:32.384 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-03-24T20:43:32.679 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2021-03-24T20:43:32.687 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-html5lib. 2021-03-24T20:43:32.696 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:43:32.705 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-03-24T20:43:32.737 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr-k8sevents (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:32.745 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-03-24T20:43:33.082 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-json-pointer. 2021-03-24T20:43:33.090 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-03-24T20:43:33.096 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../077-python3-json-pointer_1.10-1_all.deb ... 2021-03-24T20:43:33.107 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-03-24T20:43:33.140 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-json-pointer (1.10-1) ... 2021-03-24T20:43:33.148 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-24T20:43:33.510 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-jsonpatch. 2021-03-24T20:43:33.525 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../078-python3-jsonpatch_1.19+really1.16-1fakesync1_all.deb ... 2021-03-24T20:43:33.571 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-24T20:43:33.627 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pyparsing. 2021-03-24T20:43:33.641 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-03-24T20:43:33.685 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-24T20:43:33.871 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-mgr-rook. 2021-03-24T20:43:33.884 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../079-ceph-mgr-rook_15.2.10-76-gdcf77bd9-1bionic_all.deb ... 2021-03-24T20:43:33.929 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-mgr-rook (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:34.046 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-tz. 2021-03-24T20:43:34.063 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-03-24T20:43:34.113 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-tz (2018.3-2) ... 2021-03-24T20:43:34.298 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libonig4:amd64. 2021-03-24T20:43:34.315 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../080-libonig4_6.7.0-1_amd64.deb ... 2021-03-24T20:43:34.416 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-matplotlib. 2021-03-24T20:43:34.430 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-03-24T20:43:34.441 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libonig4:amd64 (6.7.0-1) ... 2021-03-24T20:43:34.474 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-24T20:43:34.794 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libjq1:amd64. 2021-03-24T20:43:34.808 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../081-libjq1_1.5+dfsg-2_amd64.deb ... 2021-03-24T20:43:34.852 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libjq1:amd64 (1.5+dfsg-2) ... 2021-03-24T20:43:35.172 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package jq. 2021-03-24T20:43:35.188 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../082-jq_1.5+dfsg-2_amd64.deb ... 2021-03-24T20:43:35.230 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking jq (1.5+dfsg-2) ... 2021-03-24T20:43:35.491 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package socat. 2021-03-24T20:43:35.506 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../083-socat_1.7.3.2-2ubuntu2_amd64.deb ... 2021-03-24T20:43:35.522 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-olefile. 2021-03-24T20:43:35.537 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-03-24T20:43:35.557 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking socat (1.7.3.2-2ubuntu2) ... 2021-03-24T20:43:35.597 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-03-24T20:43:35.892 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pastescript. 2021-03-24T20:43:35.908 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-03-24T20:43:35.936 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libxslt1.1:amd64. 2021-03-24T20:43:35.952 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../084-libxslt1.1_1.1.29-5ubuntu0.2_amd64.deb ... 2021-03-24T20:43:35.958 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-03-24T20:43:36.002 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-24T20:43:36.280 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package xmlstarlet. 2021-03-24T20:43:36.295 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../085-xmlstarlet_1.6.1-2_amd64.deb ... 2021-03-24T20:43:36.304 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pil:amd64. 2021-03-24T20:43:36.318 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.5_amd64.deb ... 2021-03-24T20:43:36.338 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking xmlstarlet (1.6.1-2) ... 2021-03-24T20:43:36.457 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.5) ... 2021-03-24T20:43:36.718 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package ceph-test. 2021-03-24T20:43:36.733 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../086-ceph-test_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:36.776 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking ceph-test (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:36.798 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pluggy. 2021-03-24T20:43:36.815 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-03-24T20:43:36.856 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-03-24T20:43:37.159 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-py. 2021-03-24T20:43:37.177 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-03-24T20:43:37.218 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-py (1.5.2-1) ... 2021-03-24T20:43:37.579 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pyinotify. 2021-03-24T20:43:37.594 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-03-24T20:43:37.646 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-03-24T20:43:37.974 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-pytest. 2021-03-24T20:43:37.988 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-03-24T20:43:38.032 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-03-24T20:43:38.402 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-simplejson. 2021-03-24T20:43:38.419 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-03-24T20:43:38.470 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-03-24T20:43:38.797 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package python3-tk:amd64. 2021-03-24T20:43:38.814 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-03-24T20:43:38.855 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-24T20:43:39.292 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package radosgw. 2021-03-24T20:43:39.306 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../111-radosgw_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:39.350 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking radosgw (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:40.087 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package libcephfs-dev. 2021-03-24T20:43:40.104 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../087-libcephfs-dev_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:40.145 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking libcephfs-dev (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:40.329 INFO:teuthology.orchestra.run.smithi052.stdout:Selecting previously unselected package rbd-fuse. 2021-03-24T20:43:40.344 INFO:teuthology.orchestra.run.smithi052.stdout:Preparing to unpack .../112-rbd-fuse_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:40.387 INFO:teuthology.orchestra.run.smithi052.stdout:Unpacking rbd-fuse (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:40.406 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package nvme-cli. 2021-03-24T20:43:40.420 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../088-nvme-cli_1.5-1ubuntu1.1_amd64.deb ... 2021-03-24T20:43:40.464 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking nvme-cli (1.5-1ubuntu1.1) ... 2021-03-24T20:43:40.695 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-03-24T20:43:40.818 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python-attr. 2021-03-24T20:43:40.834 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../089-python-attr_17.4.0-2_all.deb ... 2021-03-24T20:43:40.877 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python-attr (17.4.0-2) ... 2021-03-24T20:43:41.008 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-24T20:43:41.179 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python-funcsigs. 2021-03-24T20:43:41.196 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../090-python-funcsigs_1.0.2-4_all.deb ... 2021-03-24T20:43:41.237 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python-funcsigs (1.0.2-4) ... 2021-03-24T20:43:41.385 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-24T20:43:41.511 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-24T20:43:41.582 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python-pluggy. 2021-03-24T20:43:41.603 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../091-python-pluggy_0.6.0-1_all.deb ... 2021-03-24T20:43:41.640 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python-pluggy (0.6.0-1) ... 2021-03-24T20:43:41.943 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python-py. 2021-03-24T20:43:41.958 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../092-python-py_1.5.2-1_all.deb ... 2021-03-24T20:43:42.002 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python-py (1.5.2-1) ... 2021-03-24T20:43:42.138 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.5) ... 2021-03-24T20:43:42.354 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python-pytest. 2021-03-24T20:43:42.368 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../093-python-pytest_3.3.2-2_all.deb ... 2021-03-24T20:43:42.421 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python-pytest (3.3.2-2) ... 2021-03-24T20:43:42.523 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-03-24T20:43:42.816 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-webencodings (0.5-2) ... 2021-03-24T20:43:42.816 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-attr. 2021-03-24T20:43:42.831 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../094-python3-attr_17.4.0-2_all.deb ... 2021-03-24T20:43:42.906 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-attr (17.4.0-2) ... 2021-03-24T20:43:43.108 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-attr (17.4.0-2) ... 2021-03-24T20:43:43.210 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-cycler. 2021-03-24T20:43:43.224 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../095-python3-cycler_0.10.0-1_all.deb ... 2021-03-24T20:43:43.268 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-cycler (0.10.0-1) ... 2021-03-24T20:43:43.318 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-03-24T20:43:43.447 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-py (1.5.2-1) ... 2021-03-24T20:43:43.571 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-webencodings. 2021-03-24T20:43:43.586 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../096-python3-webencodings_0.5-2_all.deb ... 2021-03-24T20:43:43.629 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-webencodings (0.5-2) ... 2021-03-24T20:43:43.796 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-03-24T20:43:43.923 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-html5lib. 2021-03-24T20:43:43.941 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../097-python3-html5lib_0.999999999-1_all.deb ... 2021-03-24T20:43:43.983 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-html5lib (0.999999999-1) ... 2021-03-24T20:43:44.137 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-03-24T20:43:44.335 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-lxml:amd64. 2021-03-24T20:43:44.350 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../098-python3-lxml_4.2.1-1ubuntu0.3_amd64.deb ... 2021-03-24T20:43:44.401 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-24T20:43:44.432 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-03-24T20:43:44.733 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-ceph-argparse (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:44.862 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pyparsing. 2021-03-24T20:43:44.880 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../099-python3-pyparsing_2.2.0+dfsg1-2_all.deb ... 2021-03-24T20:43:44.921 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-24T20:43:45.076 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-03-24T20:43:45.274 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-tz. 2021-03-24T20:43:45.290 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../100-python3-tz_2018.3-2_all.deb ... 2021-03-24T20:43:45.332 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-tz (2018.3-2) ... 2021-03-24T20:43:45.394 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-03-24T20:43:45.520 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-24T20:43:45.626 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-matplotlib. 2021-03-24T20:43:45.640 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../101-python3-matplotlib_2.1.1-2ubuntu3_amd64.deb ... 2021-03-24T20:43:45.684 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-24T20:43:46.441 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-olefile. 2021-03-24T20:43:46.455 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../102-python3-olefile_0.45.1-1_all.deb ... 2021-03-24T20:43:46.481 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-03-24T20:43:46.499 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-olefile (0.45.1-1) ... 2021-03-24T20:43:46.799 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-03-24T20:43:46.827 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pastescript. 2021-03-24T20:43:46.840 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../103-python3-pastescript_2.0.2-2_all.deb ... 2021-03-24T20:43:46.885 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pastescript (2.0.2-2) ... 2021-03-24T20:43:47.125 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-py (1.5.2-1) ... 2021-03-24T20:43:47.205 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pil:amd64. 2021-03-24T20:43:47.221 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../104-python3-pil_5.1.0-1ubuntu0.5_amd64.deb ... 2021-03-24T20:43:47.263 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pil:amd64 (5.1.0-1ubuntu0.5) ... 2021-03-24T20:43:47.351 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-03-24T20:43:47.633 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pluggy. 2021-03-24T20:43:47.648 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../105-python3-pluggy_0.6.0-1_all.deb ... 2021-03-24T20:43:47.694 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-03-24T20:43:47.708 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pluggy (0.6.0-1) ... 2021-03-24T20:43:47.784 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-03-24T20:43:48.010 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-py. 2021-03-24T20:43:48.026 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../106-python3-py_1.5.2-1_all.deb ... 2021-03-24T20:43:48.037 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-03-24T20:43:48.069 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-py (1.5.2-1) ... 2021-03-24T20:43:48.180 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-03-24T20:43:48.373 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-03-24T20:43:48.397 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pyinotify. 2021-03-24T20:43:48.411 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../107-python3-pyinotify_0.9.6-1_all.deb ... 2021-03-24T20:43:48.455 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pyinotify (0.9.6-1) ... 2021-03-24T20:43:48.657 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-24T20:43:48.783 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-pytest. 2021-03-24T20:43:48.798 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../108-python3-pytest_3.3.2-2_all.deb ... 2021-03-24T20:43:48.841 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-pytest (3.3.2-2) ... 2021-03-24T20:43:48.992 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-03-24T20:43:49.194 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-simplejson. 2021-03-24T20:43:49.210 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../109-python3-simplejson_3.13.2-1_amd64.deb ... 2021-03-24T20:43:49.269 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-simplejson (3.13.2-1) ... 2021-03-24T20:43:49.285 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-03-24T20:43:49.669 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-03-24T20:43:49.997 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package python3-tk:amd64. 2021-03-24T20:43:50.004 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-03-24T20:43:50.011 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../110-python3-tk_3.6.9-1~18.04_amd64.deb ... 2021-03-24T20:43:50.055 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-24T20:43:50.355 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up cephadm (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:50.559 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package radosgw. 2021-03-24T20:43:50.575 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../111-radosgw_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:50.617 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking radosgw (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:51.006 INFO:teuthology.orchestra.run.smithi052.stdout:Adding system user cephadm....done 2021-03-24T20:43:51.170 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-03-24T20:43:51.455 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-24T20:43:51.579 INFO:teuthology.orchestra.run.smithi063.stdout:Selecting previously unselected package rbd-fuse. 2021-03-24T20:43:51.594 INFO:teuthology.orchestra.run.smithi063.stdout:Preparing to unpack .../112-rbd-fuse_15.2.10-76-gdcf77bd9-1bionic_amd64.deb ... 2021-03-24T20:43:51.597 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-03-24T20:43:51.637 INFO:teuthology.orchestra.run.smithi063.stdout:Unpacking rbd-fuse (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:51.757 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-03-24T20:43:51.931 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-logutils (0.3.3-5) ... 2021-03-24T20:43:52.116 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-03-24T20:43:52.233 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-werkzeug (0.14.1+dfsg1-1ubuntu0.1) ... 2021-03-24T20:43:52.426 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-03-24T20:43:52.552 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-03-24T20:43:52.635 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up librdkafka1:amd64 (0.11.3-1build1) ... 2021-03-24T20:43:52.760 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-sqlalchemy (1.1.11+ds1-1ubuntu1) ... 2021-03-24T20:43:52.903 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-ceph-common (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:53.221 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-monotonic (1.1-2) ... 2021-03-24T20:43:53.371 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pil:amd64 (5.1.0-1ubuntu0.5) ... 2021-03-24T20:43:53.539 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-24T20:43:53.665 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-24T20:43:53.772 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-simplegeneric (0.8.1-1) ... 2021-03-24T20:43:54.024 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-03-24T20:43:54.065 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-webencodings (0.5-2) ... 2021-03-24T20:43:54.366 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python-attr (17.4.0-2) ... 2021-03-24T20:43:54.392 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-24T20:43:54.491 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-03-24T20:43:54.524 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-03-24T20:43:54.593 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-cffi-backend (1.11.5-1) ... 2021-03-24T20:43:54.719 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-py (1.5.2-1) ... 2021-03-24T20:43:54.770 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-03-24T20:43:55.061 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-waitress (1.0.1-1) ... 2021-03-24T20:43:55.079 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-24T20:43:55.371 INFO:teuthology.orchestra.run.smithi063.stdout:update-alternatives: using /usr/bin/waitress-serve-python3 to provide /usr/bin/waitress-serve (waitress-serve) in auto mode 2021-03-24T20:43:55.397 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-24T20:43:55.672 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-olefile (0.45.1-1) ... 2021-03-24T20:43:55.740 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-attr (17.4.0-2) ... 2021-03-24T20:43:55.974 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-ceph-argparse (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:43:56.041 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-03-24T20:43:56.233 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-24T20:43:56.283 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-tempita (0.5.2-2) ... 2021-03-24T20:43:56.592 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up socat (1.7.3.2-2ubuntu2) ... 2021-03-24T20:43:56.652 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-03-24T20:43:56.718 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-numpy (1:1.13.3-2ubuntu1) ... 2021-03-24T20:43:56.742 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-03-24T20:43:57.012 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up liboath0 (2.6.1-1) ... 2021-03-24T20:43:57.188 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-24T20:43:57.339 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-24T20:43:57.590 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-03-24T20:43:57.679 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-webob (1:1.7.3-2fakesync1) ... 2021-03-24T20:43:57.725 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-03-24T20:43:57.989 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-bcrypt (3.1.4-2) ... 2021-03-24T20:43:58.135 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-24T20:43:58.261 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-03-24T20:43:58.290 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python-py (1.5.2-1) ... 2021-03-24T20:43:58.525 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-simplejson (3.13.2-1) ... 2021-03-24T20:43:58.562 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-tz (2018.3-2) ... 2021-03-24T20:43:58.876 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-json-pointer (1.10-1) ... 2021-03-24T20:43:58.896 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-03-24T20:43:58.966 INFO:teuthology.orchestra.run.smithi063.stdout:update-alternatives: using /usr/bin/python3-jsonpointer to provide /usr/bin/jsonpointer (jsonpointer) in auto mode 2021-03-24T20:43:59.206 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-03-24T20:43:59.210 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libonig4:amd64 (6.7.0-1) ... 2021-03-24T20:43:59.337 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python-pluggy (0.6.0-1) ... 2021-03-24T20:43:59.524 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-joblib (0.11-1) ... 2021-03-24T20:43:59.529 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-singledispatch (3.4.0.3-2) ... 2021-03-24T20:43:59.830 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pyparsing (2.2.0+dfsg1-2) ... 2021-03-24T20:43:59.875 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-routes (2.4.1-1) ... 2021-03-24T20:44:00.148 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-cycler (0.10.0-1) ... 2021-03-24T20:44:00.184 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-03-24T20:44:00.457 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-cherrypy3 (8.9.1-2) ... 2021-03-24T20:44:00.502 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-03-24T20:44:00.828 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-24T20:44:00.842 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-bs4 (4.6.0-1) ... 2021-03-24T20:44:01.193 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pluggy (0.6.0-1) ... 2021-03-24T20:44:01.213 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-03-24T20:44:01.494 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up cephadm (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:01.506 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up jq (1.5+dfsg-2) ... 2021-03-24T20:44:01.607 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-03-24T20:44:02.198 INFO:teuthology.orchestra.run.smithi063.stdout:Adding system user cephadm....done 2021-03-24T20:44:02.361 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-markupsafe (1.0-1build1) ... 2021-03-24T20:44:02.662 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libxslt1.1:amd64 (1.1.29-5ubuntu0.2) ... 2021-03-24T20:44:02.779 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-24T20:44:02.788 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up nvme-cli (1.5-1ubuntu1.1) ... 2021-03-24T20:44:02.931 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-asn1crypto (0.24.0-1) ... 2021-03-24T20:44:03.090 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-03-24T20:44:03.291 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-prettytable (0.7.2-3) ... 2021-03-24T20:44:03.398 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-03-24T20:44:03.500 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-03-24T20:44:03.592 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libjq1:amd64 (1.5+dfsg-2) ... 2021-03-24T20:44:03.735 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pyasn1 (0.4.2-3) ... 2021-03-24T20:44:03.867 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python-pytest (3.3.2-2) ... 2021-03-24T20:44:04.069 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-ceph-common (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:04.118 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-03-24T20:44:04.244 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-03-24T20:44:04.370 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up librados2 (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:04.379 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-monotonic (1.1-2) ... 2021-03-24T20:44:04.496 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-03-24T20:44:04.682 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libsnappy1v5:amd64 (1.1.7-1) ... 2021-03-24T20:44:04.797 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libcephfs2 (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:04.806 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-jinja2 (2.10-1ubuntu0.18.04.1) ... 2021-03-24T20:44:04.898 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libcephfs-dev (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:04.999 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-rados (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:05.100 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-03-24T20:44:05.157 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-html5lib (0.999999999-1) ... 2021-03-24T20:44:05.426 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-03-24T20:44:05.509 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-jsonpatch (1.19+really1.16-1fakesync1) ... 2021-03-24T20:44:05.624 INFO:teuthology.orchestra.run.smithi063.stdout:update-alternatives: using /usr/bin/python3-jsondiff to provide /usr/bin/jsondiff (jsondiff) in auto mode 2021-03-24T20:44:05.656 INFO:teuthology.orchestra.run.smithi063.stdout:update-alternatives: using /usr/bin/python3-jsonpatch to provide /usr/bin/jsonpatch (jsonpatch) in auto mode 2021-03-24T20:44:05.894 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-repoze.lru (0.7-1) ... 2021-03-24T20:44:06.119 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-03-24T20:44:06.204 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-httplib2 (0.9.2+dfsg-1ubuntu0.3) ... 2021-03-24T20:44:06.445 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-24T20:44:06.513 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2) ... 2021-03-24T20:44:06.873 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-attr (17.4.0-2) ... 2021-03-24T20:44:07.182 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python-funcsigs (1.0.2-4) ... 2021-03-24T20:44:07.383 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-paste (2.0.3+dfsg-4ubuntu1) ... 2021-03-24T20:44:07.408 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-03-24T20:44:07.802 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-websocket (0.44.0-0ubuntu2) ... 2021-03-24T20:44:07.891 INFO:teuthology.orchestra.run.smithi063.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2021-03-24T20:44:08.129 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up liboath0 (2.6.1-1) ... 2021-03-24T20:44:08.279 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-03-24T20:44:08.330 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up librabbitmq4:amd64 (0.8.0-1ubuntu0.18.04.2) ... 2021-03-24T20:44:08.456 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-tk:amd64 (3.6.9-1~18.04) ... 2021-03-24T20:44:08.580 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up libradosstriper1 (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:08.681 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-cephfs (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:08.707 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python-pastedeploy-tpl (1.5.2-4) ... 2021-03-24T20:44:08.833 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-setuptools (39.0.1-2) ... 2021-03-24T20:44:08.957 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up librgw2 (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:09.058 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-fuse (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:09.243 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up liblttng-ust0:amd64 (2.10.1-1) ... 2021-03-24T20:44:09.276 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-24T20:44:09.277 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-24T20:44:09.394 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-decorator (4.1.2-1) ... 2021-03-24T20:44:09.518 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up librbd1 (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:09.644 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-03-24T20:44:09.687 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-tz (2018.3-2) ... 2021-03-24T20:44:09.735 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-03-24T20:44:09.775 INFO:teuthology.orchestra.run.smithi052.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-03-24T20:44:09.988 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-uritemplate (0.6-1ubuntu1) ... 2021-03-24T20:44:10.089 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-24T20:44:10.281 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-jwt (1.5.3+ds1-1) ... 2021-03-24T20:44:10.591 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-joblib (0.11-1) ... 2021-03-24T20:44:10.775 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-modules-core (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:10.892 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-03-24T20:44:10.950 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-routes (2.4.1-1) ... 2021-03-24T20:44:11.219 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-rgw (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:11.268 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pyinotify (0.9.6-1) ... 2021-03-24T20:44:11.344 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up rbd-fuse (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:11.470 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up python3-rbd (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:11.596 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-common (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:11.619 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-dateutil (2.6.1-1) ... 2021-03-24T20:44:11.945 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-cryptography (2.1.4-1ubuntu1.4) ... 2021-03-24T20:44:12.047 INFO:teuthology.orchestra.run.smithi052.stdout:Adding group ceph....done 2021-03-24T20:44:12.322 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-fasteners (0.12.0-3) ... 2021-03-24T20:44:12.458 INFO:teuthology.orchestra.run.smithi052.stdout:Adding system user ceph....done 2021-03-24T20:44:12.623 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up jq (1.5+dfsg-2) ... 2021-03-24T20:44:12.650 INFO:teuthology.orchestra.run.smithi052.stdout:Setting system user ceph properties....done 2021-03-24T20:44:12.749 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-scipy (0.19.1-2ubuntu1) ... 2021-03-24T20:44:12.816 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-03-24T20:44:13.044 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-03-24T20:44:13.348 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up radosgw (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:13.592 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-24T20:44:13.592 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-24T20:44:13.958 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-test (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:13.963 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-lxml:amd64 (4.2.1-1ubuntu0.3) ... 2021-03-24T20:44:14.059 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-base (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:14.256 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-mako (1.0.7+ds1-1) ... 2021-03-24T20:44:14.398 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-03-24T20:44:14.557 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-sklearn-lib (0.19.1-3) ... 2021-03-24T20:44:14.653 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:14.675 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pytest (3.3.2-2) ... 2021-03-24T20:44:14.897 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-24T20:44:14.898 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-24T20:44:15.060 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python-pytest (3.3.2-2) ... 2021-03-24T20:44:15.154 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-diskprediction-local (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:15.305 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-osd (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:15.319 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libleveldb1v5:amd64 (1.20-2) ... 2021-03-24T20:44:15.445 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up xmlstarlet (1.6.1-2) ... 2021-03-24T20:44:15.572 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up librados2 (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:15.689 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-rsa (3.4.2-1) ... 2021-03-24T20:44:15.743 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-24T20:44:15.743 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-24T20:44:15.990 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libcephfs2 (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:16.124 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libcephfs-dev (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:16.149 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-cephadm (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:16.242 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-rados (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:16.267 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:16.368 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-openssl (17.5.0-1ubuntu1) ... 2021-03-24T20:44:16.404 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-rook (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:16.546 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mds (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:16.694 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pastedeploy (1.5.2-4) ... 2021-03-24T20:44:16.778 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-24T20:44:16.778 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-24T20:44:17.003 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-oauth2client (4.1.2-3) ... 2021-03-24T20:44:17.015 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mon (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:17.243 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-24T20:44:17.244 INFO:teuthology.orchestra.run.smithi052.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-24T20:44:17.330 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-matplotlib (2.1.1-2ubuntu3) ... 2021-03-24T20:44:17.499 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-k8sevents (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:17.600 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph-mgr-dashboard (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:17.726 INFO:teuthology.orchestra.run.smithi052.stdout:Setting up ceph (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:17.852 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-03-24T20:44:18.045 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-03-24T20:44:18.293 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-sklearn (0.19.1-3) ... 2021-03-24T20:44:19.148 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-webtest (2.0.28-1ubuntu1) ... 2021-03-24T20:44:19.415 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up libradosstriper1 (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:19.516 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-cephfs (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:19.800 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up librgw2 (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:19.935 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-fuse (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:20.131 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-03-24T20:44:20.174 INFO:teuthology.orchestra.run.smithi052.stdout:ureadahead will be reprofiled on next reboot 2021-03-24T20:44:20.231 INFO:teuthology.orchestra.run.smithi052.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-03-24T20:44:20.436 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-24T20:44:20.437 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2021-03-24T20:44:20.680 INFO:teuthology.orchestra.run.smithi052.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-03-24T20:44:20.683 DEBUG:teuthology.parallel:result is None 2021-03-24T20:44:20.744 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up librbd1 (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:20.887 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pecan (1.2.1-2) ... 2021-03-24T20:44:20.977 INFO:teuthology.orchestra.run.smithi063.stdout:update-alternatives: using /usr/bin/python3-pecan to provide /usr/bin/pecan (pecan) in auto mode 2021-03-24T20:44:21.018 INFO:teuthology.orchestra.run.smithi063.stdout:update-alternatives: using /usr/bin/python3-gunicorn_pecan to provide /usr/bin/gunicorn_pecan (gunicorn_pecan) in auto mode 2021-03-24T20:44:21.316 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-kubernetes (2.0.0-2ubuntu1) ... 2021-03-24T20:44:21.978 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr-modules-core (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:22.102 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-pastescript (2.0.2-2) ... 2021-03-24T20:44:22.454 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-rgw (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:22.579 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up rbd-fuse (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:22.706 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up python3-rbd (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:22.831 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-common (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:23.307 INFO:teuthology.orchestra.run.smithi063.stdout:Adding group ceph....done 2021-03-24T20:44:23.693 INFO:teuthology.orchestra.run.smithi063.stdout:Adding system user ceph....done 2021-03-24T20:44:23.894 INFO:teuthology.orchestra.run.smithi063.stdout:Setting system user ceph properties....done 2021-03-24T20:44:24.061 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2021-03-24T20:44:24.283 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2021-03-24T20:44:24.583 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up radosgw (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:24.830 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-24T20:44:24.831 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2021-03-24T20:44:25.260 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-test (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:25.361 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-base (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:25.847 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2021-03-24T20:44:26.155 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:26.384 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-24T20:44:26.385 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2021-03-24T20:44:26.657 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr-diskprediction-local (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:26.799 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-osd (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:27.260 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-24T20:44:27.260 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2021-03-24T20:44:27.543 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr-cephadm (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:27.661 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:27.804 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr-rook (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:27.946 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mds (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:28.174 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-24T20:44:28.175 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2021-03-24T20:44:28.464 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mon (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:28.682 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-24T20:44:28.683 INFO:teuthology.orchestra.run.smithi063.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2021-03-24T20:44:29.016 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr-k8sevents (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:29.142 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph-mgr-dashboard (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:29.268 INFO:teuthology.orchestra.run.smithi063.stdout:Setting up ceph (15.2.10-76-gdcf77bd9-1bionic) ... 2021-03-24T20:44:29.369 INFO:teuthology.orchestra.run.smithi063.stdout:Processing triggers for systemd (237-3ubuntu10.44) ... 2021-03-24T20:44:29.553 INFO:teuthology.orchestra.run.smithi063.stdout:Processing triggers for man-db (2.8.3-2ubuntu0.1) ... 2021-03-24T20:44:31.640 INFO:teuthology.orchestra.run.smithi063.stdout:Processing triggers for ureadahead (0.100.0-21) ... 2021-03-24T20:44:31.683 INFO:teuthology.orchestra.run.smithi063.stdout:ureadahead will be reprofiled on next reboot 2021-03-24T20:44:31.741 INFO:teuthology.orchestra.run.smithi063.stdout:Processing triggers for libc-bin (2.27-3ubuntu1.4) ... 2021-03-24T20:44:32.184 INFO:teuthology.orchestra.run.smithi063.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-03-24T20:44:32.187 DEBUG:teuthology.parallel:result is None 2021-03-24T20:44:32.188 DEBUG:teuthology.orchestra.run.smithi052:> sudo lsb_release -is 2021-03-24T20:44:32.245 INFO:teuthology.orchestra.run.smithi052.stdout:Ubuntu 2021-03-24T20:44:32.250 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-24T20:44:32.251 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-03-24T20:44:32.251 INFO:teuthology.packaging:ref: None 2021-03-24T20:44:32.251 INFO:teuthology.packaging:tag: None 2021-03-24T20:44:32.252 INFO:teuthology.packaging:branch: octopus 2021-03-24T20:44:32.252 INFO:teuthology.packaging:sha1: 9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2 2021-03-24T20:44:32.253 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=octopus 2021-03-24T20:44:32.472 DEBUG:teuthology.orchestra.run.smithi052:> dpkg-query -W -f '${Version}' ceph 2021-03-24T20:44:32.496 INFO:teuthology.orchestra.run.smithi052.stdout:15.2.10-76-gdcf77bd9-1bionic 2021-03-24T20:44:32.497 INFO:teuthology.packaging:The installed version of ceph is 15.2.10-76-gdcf77bd9-1bionic 2021-03-24T20:44:32.498 INFO:teuthology.task.install:The correct ceph version 15.2.10-76-gdcf77bd9-1bionic is installed. 2021-03-24T20:44:32.498 DEBUG:teuthology.orchestra.run.smithi063:> sudo lsb_release -is 2021-03-24T20:44:32.555 INFO:teuthology.orchestra.run.smithi063.stdout:Ubuntu 2021-03-24T20:44:32.560 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-24T20:44:32.561 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-03-24T20:44:32.561 INFO:teuthology.packaging:ref: None 2021-03-24T20:44:32.562 INFO:teuthology.packaging:tag: None 2021-03-24T20:44:32.562 INFO:teuthology.packaging:branch: octopus 2021-03-24T20:44:32.563 INFO:teuthology.packaging:sha1: 9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2 2021-03-24T20:44:32.564 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=octopus 2021-03-24T20:44:32.783 DEBUG:teuthology.orchestra.run.smithi063:> dpkg-query -W -f '${Version}' ceph 2021-03-24T20:44:32.808 INFO:teuthology.orchestra.run.smithi063.stdout:15.2.10-76-gdcf77bd9-1bionic 2021-03-24T20:44:32.808 INFO:teuthology.packaging:The installed version of ceph is 15.2.10-76-gdcf77bd9-1bionic 2021-03-24T20:44:32.809 INFO:teuthology.task.install:The correct ceph version 15.2.10-76-gdcf77bd9-1bionic is installed. 2021-03-24T20:44:32.809 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-03-24T20:44:32.811 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:44:32.812 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-24T20:44:32.828 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:44:32.829 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-03-24T20:44:32.863 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-03-24T20:44:32.866 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:44:32.867 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/daemon-helper 2021-03-24T20:44:32.884 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-24T20:44:32.940 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:44:32.941 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/daemon-helper 2021-03-24T20:44:32.957 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-03-24T20:44:33.016 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-03-24T20:44:33.017 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:44:33.017 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-24T20:44:33.035 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-24T20:44:33.092 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:44:33.092 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/usr/bin/adjust-ulimits 2021-03-24T20:44:33.108 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-03-24T20:44:33.167 INFO:teuthology.run_tasks:Running task ceph... 2021-03-24T20:44:33.259 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-03-24T20:44:33.260 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 777 /var/log/ceph 2021-03-24T20:44:33.263 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 777 /var/log/ceph 2021-03-24T20:44:33.278 INFO:tasks.ceph:Disabling ceph logrotate... 2021-03-24T20:44:33.278 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/logrotate.d/ceph 2021-03-24T20:44:33.320 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/logrotate.d/ceph 2021-03-24T20:44:33.413 INFO:tasks.ceph:Creating extra log directories... 2021-03-24T20:44:33.414 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-03-24T20:44:33.443 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-03-24T20:44:33.556 INFO:tasks.ceph:Creating ceph cluster c1... 2021-03-24T20:44:33.556 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-03-24T20:44:33.556 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi/5994655', 'branch': 'wip-sage-testing-2021-03-24-1139', 'description': 'upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary}', 'email': None, 'first_in_suite': False, 'job_id': '5994655', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi', 'nuke-on-error': True, 'overrides': {'admin_socket': {'branch': 'wip-sage-testing-2021-03-24-1139'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'branch': 'octopus', 'sha1': '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py'], 'branch': 'octopus', 'repo': 'https://github.com/ceph/ceph.git'}, 'workunit': {'branch': 'wip-sage-testing-2021-03-24-1139', 'sha1': '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2'}}, 'owner': 'scheduled_sage@teuthology', 'priority': 75, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2', 'sleep_before_teardown': 0, 'suite': 'upgrade:octopus-x', 'suite_branch': 'wip-sage-testing-2021-03-24-1139', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2', 'targets': {'smithi052.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdmeEuI/ElAp8wa3lVeLfXYahy/pX0c/EdGRCcDqpaRhyuHrjHU0PEe8ckqbf14SelozcIfJk0CStTIKUBckWCPM5LZwtXt7Vh7NQCk6AHxWQq1egfGetVeWbzuPWS+q+doQ+4EyNnnZyWjnIXPtJc4R4tr/mCnKYWRQqax2ypu7oGzF/XCnsQeiRDdY1LWNxobRIpSIEGCJfD4NgJ1MSC4TrgjAw95hrAZgI480tzXReYyKL+WsdNrH/M4fVdlNXXNn5InUbzLyLYKaUfOkbTzr0Jp6SyzBKkBzcCg7kivZwOCv/HNZuHjh3ToUWZValSR3cjBMiSadmLcUBWcQNN', 'smithi063.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJpxVWP+MN/3YlQybedqnQLiHHtDwzkWm82LhqQYZMkhGsE3UDKFv4wmJiTlzH9DcdOW55ds5k0h8KWkxBwtGUf2Rpz3o1MYiLkHk7T/pSG1CC02Iwy7Mt8Je0N3HebLVwXGmBUDu+VIuo8CZNk3dW2KYmWISpqWkywv3/IsA4aMBtRnpYGISapE0Y67MEsM7thcPSAtroKsTa4rZnYbhySBlufPLGKlHn10I53dR5brooEVN4sfHXKATbKSUmG7DxsJOWhvu3zzR+3RSZ/aS8r40WQE5wdKjMt6n6MASnDB2nX/ztpeZK1EebhPzMoK6EhSudrfaq/cOGBB2y7bTL'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2'}}, {'parallel': ['upgrade-task']}, {'rgw': {'c1.client.0': {'port': 8000}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '6b3150e9e0aa7ca432e26f31d87920ebd77f3708', 'timestamp': '2021-03-24_19:49:41', 'tube': 'smithi', 'upgrade-task': [{'install.upgrade': {'c2.client.0': None, 'c2.client.1': None, 'c2.mgr.x': None, 'c2.mon.a': None, 'c2.osd.0': None, 'c2.osd.1': None, 'c2.osd.2': None}}], 'user': 'sage', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.9855'} 2021-03-24T20:44:33.559 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-03-24T20:44:33.562 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/c1.data 2021-03-24T20:44:33.570 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/run/ceph 2021-03-24T20:44:33.612 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/run/ceph 2021-03-24T20:44:33.632 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:44:33.673 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/scratch_devs of=/dev/stdout 2021-03-24T20:44:33.680 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-24T20:44:33.681 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-24T20:44:33.728 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-24T20:44:33.729 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-24T20:44:33.729 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 340 Links: 1 2021-03-24T20:44:33.729 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-24T20:44:33.730 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2021-03-24 20:41:53.921211055 +0000 2021-03-24T20:44:33.730 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2021-03-24 20:41:53.813157058 +0000 2021-03-24T20:44:33.730 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2021-03-24 20:41:53.813157058 +0000 2021-03-24T20:44:33.730 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2021-03-24T20:44:33.738 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2021-03-24T20:44:33.738 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2021-03-24T20:44:33.738 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000297176 s, 1.7 MB/s 2021-03-24T20:44:33.742 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-24T20:44:33.793 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-24T20:44:33.793 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-24T20:44:33.794 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 347 Links: 1 2021-03-24T20:44:33.794 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-24T20:44:33.794 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2021-03-24 20:41:54.149325051 +0000 2021-03-24T20:44:33.794 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2021-03-24 20:41:54.017259053 +0000 2021-03-24T20:44:33.794 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2021-03-24 20:41:54.017259053 +0000 2021-03-24T20:44:33.795 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2021-03-24T20:44:33.803 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2021-03-24T20:44:33.803 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2021-03-24T20:44:33.803 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000281834 s, 1.8 MB/s 2021-03-24T20:44:33.808 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-24T20:44:33.857 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-24T20:44:33.857 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-24T20:44:33.857 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 353 Links: 1 2021-03-24T20:44:33.858 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-24T20:44:33.858 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2021-03-24 20:41:54.377439045 +0000 2021-03-24T20:44:33.858 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2021-03-24 20:41:54.265383047 +0000 2021-03-24T20:44:33.858 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2021-03-24 20:41:54.265383047 +0000 2021-03-24T20:44:33.858 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2021-03-24T20:44:33.866 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2021-03-24T20:44:33.867 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2021-03-24T20:44:33.867 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000263007 s, 1.9 MB/s 2021-03-24T20:44:33.871 DEBUG:teuthology.orchestra.run.smithi052:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-24T20:44:33.920 INFO:teuthology.orchestra.run.smithi052.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-24T20:44:33.921 INFO:teuthology.orchestra.run.smithi052.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-24T20:44:33.921 INFO:teuthology.orchestra.run.smithi052.stdout:Device: 6h/6d Inode: 359 Links: 1 2021-03-24T20:44:33.921 INFO:teuthology.orchestra.run.smithi052.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-24T20:44:33.921 INFO:teuthology.orchestra.run.smithi052.stdout:Access: 2021-03-24 20:41:54.641571040 +0000 2021-03-24T20:44:33.922 INFO:teuthology.orchestra.run.smithi052.stdout:Modify: 2021-03-24 20:41:54.509505042 +0000 2021-03-24T20:44:33.922 INFO:teuthology.orchestra.run.smithi052.stdout:Change: 2021-03-24 20:41:54.509505042 +0000 2021-03-24T20:44:33.922 INFO:teuthology.orchestra.run.smithi052.stdout: Birth: - 2021-03-24T20:44:33.929 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records in 2021-03-24T20:44:33.930 INFO:teuthology.orchestra.run.smithi052.stderr:1+0 records out 2021-03-24T20:44:33.930 INFO:teuthology.orchestra.run.smithi052.stderr:512 bytes copied, 0.000278404 s, 1.8 MB/s 2021-03-24T20:44:33.934 INFO:tasks.ceph:osd dev map: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-24T20:44:33.935 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-03-24T20:44:33.935 INFO:tasks.ceph:Generating config... 2021-03-24T20:44:33.937 INFO:tasks.ceph:[client] debug rgw = 20 2021-03-24T20:44:33.938 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-03-24T20:44:33.938 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-03-24T20:44:33.938 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-24T20:44:33.939 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-03-24T20:44:33.939 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-03-24T20:44:33.939 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-03-24T20:44:33.939 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-03-24T20:44:33.940 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-03-24T20:44:33.940 INFO:tasks.ceph:[client] setgroup = ceph 2021-03-24T20:44:33.940 INFO:tasks.ceph:[client] setuser = ceph 2021-03-24T20:44:33.941 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-03-24T20:44:33.941 INFO:tasks.ceph:[mgr] debug ms = 1 2021-03-24T20:44:33.941 INFO:tasks.ceph:[mon] debug mon = 20 2021-03-24T20:44:33.941 INFO:tasks.ceph:[mon] debug ms = 1 2021-03-24T20:44:33.942 INFO:tasks.ceph:[mon] debug paxos = 20 2021-03-24T20:44:33.942 INFO:tasks.ceph:[osd] debug ms = 1 2021-03-24T20:44:33.942 INFO:tasks.ceph:[osd] debug osd = 20 2021-03-24T20:44:33.943 INFO:tasks.ceph:Setting up c1.mon.a... 2021-03-24T20:44:33.943 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c1.keyring 2021-03-24T20:44:34.010 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/c1.keyring 2021-03-24T20:44:34.012 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c1.keyring 2021-03-24T20:44:34.100 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-03-24T20:44:34.163 DEBUG:tasks.ceph:Ceph mon addresses: [('c1.mon.a', '172.21.15.52')] 2021-03-24T20:44:34.164 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.52'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}} 2021-03-24T20:44:34.165 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:44:34.166 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-03-24T20:44:34.224 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.52 --print /home/ubuntu/cephtest/c1.monmap 2021-03-24T20:44:34.305 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:44:34.306 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:44:34.307 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c1.monmap 2021-03-24T20:44:34.307 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: generated fsid 91c6d487-aa59-4b0e-bf1a-56b2e36b57d7 2021-03-24T20:44:34.307 INFO:teuthology.orchestra.run.smithi052.stdout:epoch 0 2021-03-24T20:44:34.308 INFO:teuthology.orchestra.run.smithi052.stdout:fsid 91c6d487-aa59-4b0e-bf1a-56b2e36b57d7 2021-03-24T20:44:34.308 INFO:teuthology.orchestra.run.smithi052.stdout:last_changed 2021-03-24T20:44:34.305241+0000 2021-03-24T20:44:34.308 INFO:teuthology.orchestra.run.smithi052.stdout:created 2021-03-24T20:44:34.305241+0000 2021-03-24T20:44:34.308 INFO:teuthology.orchestra.run.smithi052.stdout:min_mon_release 0 (unknown) 2021-03-24T20:44:34.309 INFO:teuthology.orchestra.run.smithi052.stdout:0: [v2:172.21.15.52:3300/0,v1:172.21.15.52:6789/0] mon.a 2021-03-24T20:44:34.309 INFO:teuthology.orchestra.run.smithi052.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c1.monmap (1 monitors) 2021-03-24T20:44:34.310 DEBUG:teuthology.orchestra.run.smithi052:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-03-24T20:44:34.367 INFO:tasks.ceph:Writing /etc/ceph/c1.conf for FSID 91c6d487-aa59-4b0e-bf1a-56b2e36b57d7... 2021-03-24T20:44:34.368 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-03-24T20:44:34.418 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c1.conf && sudo chmod 0644 /etc/ceph/c1.conf > /dev/null 2021-03-24T20:44:34.447 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2021-03-24T20:44:34.448 INFO:teuthology.orchestra.run.smithi052.stdout: chdir = "" 2021-03-24T20:44:34.448 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-24T20:44:34.448 INFO:teuthology.orchestra.run.smithi052.stdout: auth supported = cephx 2021-03-24T20:44:34.448 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.448 INFO:teuthology.orchestra.run.smithi052.stdout: filestore xattr use omap = true 2021-03-24T20:44:34.449 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.449 INFO:teuthology.orchestra.run.smithi052.stdout: mon clock drift allowed = 1.000 2021-03-24T20:44:34.449 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.449 INFO:teuthology.orchestra.run.smithi052.stdout: osd crush chooseleaf type = 0 2021-03-24T20:44:34.450 INFO:teuthology.orchestra.run.smithi052.stdout: auth debug = true 2021-03-24T20:44:34.450 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.450 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on old message = true 2021-03-24T20:44:34.450 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on bug = true 2021-03-24T20:44:34.450 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.451 INFO:teuthology.orchestra.run.smithi052.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-24T20:44:34.451 INFO:teuthology.orchestra.run.smithi052.stdout: mon pg warn max object skew = 0 2021-03-24T20:44:34.451 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.451 INFO:teuthology.orchestra.run.smithi052.stdout: # disable pg_autoscaler by default for new pools 2021-03-24T20:44:34.452 INFO:teuthology.orchestra.run.smithi052.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-24T20:44:34.452 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.452 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default size = 2 2021-03-24T20:44:34.452 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.452 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow primary affinity = true 2021-03-24T20:44:34.453 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow pg remap = true 2021-03-24T20:44:34.453 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on legacy crush tunables = false 2021-03-24T20:44:34.453 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on crush straw calc version zero = false 2021-03-24T20:44:34.453 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on no sortbitwise = false 2021-03-24T20:44:34.453 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on osd down out interval zero = false 2021-03-24T20:44:34.454 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on too few osds = false 2021-03-24T20:44:34.454 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-24T20:44:34.454 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-24T20:44:34.454 INFO:teuthology.orchestra.run.smithi052.stdout: mon_allow_pool_size_one = true 2021-03-24T20:44:34.455 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.455 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-24T20:44:34.455 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.455 INFO:teuthology.orchestra.run.smithi052.stdout: osd default data pool replay window = 5 2021-03-24T20:44:34.455 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.456 INFO:teuthology.orchestra.run.smithi052.stdout: mon allow pool delete = true 2021-03-24T20:44:34.456 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.456 INFO:teuthology.orchestra.run.smithi052.stdout: mon cluster log file level = debug 2021-03-24T20:44:34.456 INFO:teuthology.orchestra.run.smithi052.stdout: debug asserts on shutdown = true 2021-03-24T20:44:34.456 INFO:teuthology.orchestra.run.smithi052.stdout: mon health detail to clog = false 2021-03-24T20:44:34.457 INFO:teuthology.orchestra.run.smithi052.stdout: mon host = 172.21.15.52 2021-03-24T20:44:34.457 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = 91c6d487-aa59-4b0e-bf1a-56b2e36b57d7 2021-03-24T20:44:34.457 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.457 INFO:teuthology.orchestra.run.smithi052.stdout:[osd] 2021-03-24T20:44:34.458 INFO:teuthology.orchestra.run.smithi052.stdout: osd journal size = 100 2021-03-24T20:44:34.458 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.458 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub load threshold = 5.0 2021-03-24T20:44:34.458 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub max interval = 600 2021-03-24T20:44:34.458 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.459 INFO:teuthology.orchestra.run.smithi052.stdout: osd recover clone overlap = true 2021-03-24T20:44:34.459 INFO:teuthology.orchestra.run.smithi052.stdout: osd recovery max chunk = 1048576 2021-03-24T20:44:34.459 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.460 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug shutdown = true 2021-03-24T20:44:34.460 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug op order = true 2021-03-24T20:44:34.460 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify stray on activate = true 2021-03-24T20:44:34.460 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.460 INFO:teuthology.orchestra.run.smithi052.stdout: osd open classes on start = true 2021-03-24T20:44:34.461 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug pg log writeout = true 2021-03-24T20:44:34.461 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.461 INFO:teuthology.orchestra.run.smithi052.stdout: osd deep scrub update digest min age = 30 2021-03-24T20:44:34.461 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.461 INFO:teuthology.orchestra.run.smithi052.stdout: osd map max advance = 10 2021-03-24T20:44:34.462 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.462 INFO:teuthology.orchestra.run.smithi052.stdout: journal zero on create = true 2021-03-24T20:44:34.462 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.462 INFO:teuthology.orchestra.run.smithi052.stdout: filestore ondisk finisher threads = 3 2021-03-24T20:44:34.463 INFO:teuthology.orchestra.run.smithi052.stdout: filestore apply finisher threads = 3 2021-03-24T20:44:34.463 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.463 INFO:teuthology.orchestra.run.smithi052.stdout: bdev debug aio = true 2021-03-24T20:44:34.463 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug misdirected ops = true 2021-03-24T20:44:34.464 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2021-03-24T20:44:34.464 INFO:teuthology.orchestra.run.smithi052.stdout: debug osd = 20 2021-03-24T20:44:34.464 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.464 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr] 2021-03-24T20:44:34.464 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2021-03-24T20:44:34.465 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgr = 20 2021-03-24T20:44:34.465 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2021-03-24T20:44:34.465 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2021-03-24T20:44:34.465 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2021-03-24T20:44:34.465 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2021-03-24T20:44:34.466 INFO:teuthology.orchestra.run.smithi052.stdout: mgr/telemetry/nag = false 2021-03-24T20:44:34.466 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.466 INFO:teuthology.orchestra.run.smithi052.stdout:[mon] 2021-03-24T20:44:34.466 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2021-03-24T20:44:34.466 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2021-03-24T20:44:34.467 INFO:teuthology.orchestra.run.smithi052.stdout: debug paxos = 20 2021-03-24T20:44:34.467 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2021-03-24T20:44:34.467 INFO:teuthology.orchestra.run.smithi052.stdout: mon data avail warn = 5 2021-03-24T20:44:34.467 INFO:teuthology.orchestra.run.smithi052.stdout: mon mgr mkfs grace = 240 2021-03-24T20:44:34.468 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2021-03-24T20:44:34.468 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd reporter subtree level = osd 2021-03-24T20:44:34.468 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd prime pg temp = true 2021-03-24T20:44:34.468 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2021-03-24T20:44:34.468 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:34.469 INFO:teuthology.orchestra.run.smithi052.stdout:[client] 2021-03-24T20:44:34.469 INFO:teuthology.orchestra.run.smithi052.stdout: rgw cache enabled = true 2021-03-24T20:44:34.469 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable ops log = true 2021-03-24T20:44:34.469 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable usage log = true 2021-03-24T20:44:34.469 INFO:teuthology.orchestra.run.smithi052.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-24T20:44:34.470 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-24T20:44:34.470 INFO:teuthology.orchestra.run.smithi052.stdout: debug rgw = 20 2021-03-24T20:44:34.470 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt require ssl = False 2021-03-24T20:44:34.470 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt s3 kms backend = testing 2021-03-24T20:44:34.471 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-24T20:44:34.471 INFO:teuthology.orchestra.run.smithi052.stdout: rgw curl low speed time = 300 2021-03-24T20:44:34.471 INFO:teuthology.orchestra.run.smithi052.stdout: rgw data log num shards = 4 2021-03-24T20:44:34.471 INFO:teuthology.orchestra.run.smithi052.stdout: rgw md log max shards = 4 2021-03-24T20:44:34.471 INFO:teuthology.orchestra.run.smithi052.stdout: rgw sync log trim interval = 0 2021-03-24T20:44:34.472 INFO:teuthology.orchestra.run.smithi052.stdout: rgw sync obj etag verify = True 2021-03-24T20:44:34.472 INFO:teuthology.orchestra.run.smithi052.stdout: setgroup = ceph 2021-03-24T20:44:34.472 INFO:teuthology.orchestra.run.smithi052.stdout: setuser = ceph 2021-03-24T20:44:34.472 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.a] 2021-03-24T20:44:34.474 INFO:teuthology.orchestra.run.smithi063.stdout:[global] 2021-03-24T20:44:34.474 INFO:teuthology.orchestra.run.smithi063.stdout: chdir = "" 2021-03-24T20:44:34.475 INFO:teuthology.orchestra.run.smithi063.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-24T20:44:34.475 INFO:teuthology.orchestra.run.smithi063.stdout: auth supported = cephx 2021-03-24T20:44:34.475 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.475 INFO:teuthology.orchestra.run.smithi063.stdout: filestore xattr use omap = true 2021-03-24T20:44:34.476 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.476 INFO:teuthology.orchestra.run.smithi063.stdout: mon clock drift allowed = 1.000 2021-03-24T20:44:34.476 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.476 INFO:teuthology.orchestra.run.smithi063.stdout: osd crush chooseleaf type = 0 2021-03-24T20:44:34.476 INFO:teuthology.orchestra.run.smithi063.stdout: auth debug = true 2021-03-24T20:44:34.477 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.477 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on old message = true 2021-03-24T20:44:34.477 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on bug = true 2021-03-24T20:44:34.477 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.477 INFO:teuthology.orchestra.run.smithi063.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-24T20:44:34.478 INFO:teuthology.orchestra.run.smithi063.stdout: mon pg warn max object skew = 0 2021-03-24T20:44:34.478 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.478 INFO:teuthology.orchestra.run.smithi063.stdout: # disable pg_autoscaler by default for new pools 2021-03-24T20:44:34.478 INFO:teuthology.orchestra.run.smithi063.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-24T20:44:34.479 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.479 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default size = 2 2021-03-24T20:44:34.479 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.479 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow primary affinity = true 2021-03-24T20:44:34.479 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow pg remap = true 2021-03-24T20:44:34.480 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on legacy crush tunables = false 2021-03-24T20:44:34.480 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on crush straw calc version zero = false 2021-03-24T20:44:34.480 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on no sortbitwise = false 2021-03-24T20:44:34.480 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on osd down out interval zero = false 2021-03-24T20:44:34.480 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on too few osds = false 2021-03-24T20:44:34.481 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-24T20:44:34.481 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-24T20:44:34.481 INFO:teuthology.orchestra.run.smithi063.stdout: mon_allow_pool_size_one = true 2021-03-24T20:44:34.481 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.482 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-24T20:44:34.482 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.482 INFO:teuthology.orchestra.run.smithi063.stdout: osd default data pool replay window = 5 2021-03-24T20:44:34.482 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.482 INFO:teuthology.orchestra.run.smithi063.stdout: mon allow pool delete = true 2021-03-24T20:44:34.483 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.483 INFO:teuthology.orchestra.run.smithi063.stdout: mon cluster log file level = debug 2021-03-24T20:44:34.483 INFO:teuthology.orchestra.run.smithi063.stdout: debug asserts on shutdown = true 2021-03-24T20:44:34.483 INFO:teuthology.orchestra.run.smithi063.stdout: mon health detail to clog = false 2021-03-24T20:44:34.483 INFO:teuthology.orchestra.run.smithi063.stdout: mon host = 172.21.15.52 2021-03-24T20:44:34.484 INFO:teuthology.orchestra.run.smithi063.stdout: fsid = 91c6d487-aa59-4b0e-bf1a-56b2e36b57d7 2021-03-24T20:44:34.484 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.484 INFO:teuthology.orchestra.run.smithi063.stdout:[osd] 2021-03-24T20:44:34.484 INFO:teuthology.orchestra.run.smithi063.stdout: osd journal size = 100 2021-03-24T20:44:34.484 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.485 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub load threshold = 5.0 2021-03-24T20:44:34.485 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub max interval = 600 2021-03-24T20:44:34.485 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.485 INFO:teuthology.orchestra.run.smithi063.stdout: osd recover clone overlap = true 2021-03-24T20:44:34.486 INFO:teuthology.orchestra.run.smithi063.stdout: osd recovery max chunk = 1048576 2021-03-24T20:44:34.486 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.486 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug shutdown = true 2021-03-24T20:44:34.486 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug op order = true 2021-03-24T20:44:34.486 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify stray on activate = true 2021-03-24T20:44:34.487 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.487 INFO:teuthology.orchestra.run.smithi063.stdout: osd open classes on start = true 2021-03-24T20:44:34.487 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug pg log writeout = true 2021-03-24T20:44:34.487 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.487 INFO:teuthology.orchestra.run.smithi063.stdout: osd deep scrub update digest min age = 30 2021-03-24T20:44:34.488 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.488 INFO:teuthology.orchestra.run.smithi063.stdout: osd map max advance = 10 2021-03-24T20:44:34.488 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.488 INFO:teuthology.orchestra.run.smithi063.stdout: journal zero on create = true 2021-03-24T20:44:34.489 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.489 INFO:teuthology.orchestra.run.smithi063.stdout: filestore ondisk finisher threads = 3 2021-03-24T20:44:34.489 INFO:teuthology.orchestra.run.smithi063.stdout: filestore apply finisher threads = 3 2021-03-24T20:44:34.489 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.489 INFO:teuthology.orchestra.run.smithi063.stdout: bdev debug aio = true 2021-03-24T20:44:34.490 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug misdirected ops = true 2021-03-24T20:44:34.490 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2021-03-24T20:44:34.490 INFO:teuthology.orchestra.run.smithi063.stdout: debug osd = 20 2021-03-24T20:44:34.490 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.490 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr] 2021-03-24T20:44:34.491 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2021-03-24T20:44:34.491 INFO:teuthology.orchestra.run.smithi063.stdout: debug mgr = 20 2021-03-24T20:44:34.491 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2021-03-24T20:44:34.491 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2021-03-24T20:44:34.491 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2021-03-24T20:44:34.492 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2021-03-24T20:44:34.492 INFO:teuthology.orchestra.run.smithi063.stdout: mgr/telemetry/nag = false 2021-03-24T20:44:34.492 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.492 INFO:teuthology.orchestra.run.smithi063.stdout:[mon] 2021-03-24T20:44:34.493 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2021-03-24T20:44:34.493 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2021-03-24T20:44:34.493 INFO:teuthology.orchestra.run.smithi063.stdout: debug paxos = 20 2021-03-24T20:44:34.493 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2021-03-24T20:44:34.493 INFO:teuthology.orchestra.run.smithi063.stdout: mon data avail warn = 5 2021-03-24T20:44:34.494 INFO:teuthology.orchestra.run.smithi063.stdout: mon mgr mkfs grace = 240 2021-03-24T20:44:34.494 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2021-03-24T20:44:34.494 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd reporter subtree level = osd 2021-03-24T20:44:34.494 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd prime pg temp = true 2021-03-24T20:44:34.494 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2021-03-24T20:44:34.495 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:44:34.495 INFO:teuthology.orchestra.run.smithi063.stdout:[client] 2021-03-24T20:44:34.495 INFO:teuthology.orchestra.run.smithi063.stdout: rgw cache enabled = true 2021-03-24T20:44:34.495 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable ops log = true 2021-03-24T20:44:34.495 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable usage log = true 2021-03-24T20:44:34.496 INFO:teuthology.orchestra.run.smithi063.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-24T20:44:34.496 INFO:teuthology.orchestra.run.smithi063.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-24T20:44:34.496 INFO:teuthology.orchestra.run.smithi063.stdout: debug rgw = 20 2021-03-24T20:44:34.496 INFO:teuthology.orchestra.run.smithi063.stdout: rgw crypt require ssl = False 2021-03-24T20:44:34.497 INFO:teuthology.orchestra.run.smithi063.stdout: rgw crypt s3 kms backend = testing 2021-03-24T20:44:34.497 INFO:teuthology.orchestra.run.smithi063.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-24T20:44:34.497 INFO:teuthology.orchestra.run.smithi063.stdout: rgw curl low speed time = 300 2021-03-24T20:44:34.497 INFO:teuthology.orchestra.run.smithi063.stdout: rgw data log num shards = 4 2021-03-24T20:44:34.497 INFO:teuthology.orchestra.run.smithi063.stdout: rgw md log max shards = 4 2021-03-24T20:44:34.498 INFO:teuthology.orchestra.run.smithi063.stdout: rgw sync log trim interval = 0 2021-03-24T20:44:34.498 INFO:teuthology.orchestra.run.smithi063.stdout: rgw sync obj etag verify = True 2021-03-24T20:44:34.498 INFO:teuthology.orchestra.run.smithi063.stdout: setgroup = ceph 2021-03-24T20:44:34.498 INFO:teuthology.orchestra.run.smithi063.stdout: setuser = ceph 2021-03-24T20:44:34.498 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.a] 2021-03-24T20:44:34.499 INFO:tasks.ceph:Creating admin key on c1.mon.a... 2021-03-24T20:44:34.499 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c1.keyring 2021-03-24T20:44:34.571 INFO:tasks.ceph:Copying monmap to all nodes... 2021-03-24T20:44:34.572 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:44:34.572 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/c1.keyring of=/dev/stdout 2021-03-24T20:44:34.626 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:44:34.627 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/home/ubuntu/cephtest/c1.monmap of=/dev/stdout 2021-03-24T20:44:34.678 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi052.front.sepia.ceph.com 2021-03-24T20:44:34.678 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:44:34.679 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/c1.keyring 2021-03-24T20:44:34.679 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-03-24T20:44:34.748 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:44:34.748 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-03-24T20:44:34.796 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi063.front.sepia.ceph.com 2021-03-24T20:44:34.796 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:44:34.797 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/c1.keyring 2021-03-24T20:44:34.797 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/c1.keyring 2021-03-24T20:44:34.823 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:44:34.823 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/c1.monmap 2021-03-24T20:44:34.871 INFO:tasks.ceph:Setting up mon nodes... 2021-03-24T20:44:34.872 INFO:tasks.ceph:Setting up mgr nodes... 2021-03-24T20:44:34.872 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mgr/c1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c1-x/keyring 2021-03-24T20:44:34.918 INFO:teuthology.orchestra.run.smithi052.stdout:creating /var/lib/ceph/mgr/c1-x/keyring 2021-03-24T20:44:34.919 INFO:tasks.ceph:Setting up mds nodes... 2021-03-24T20:44:34.920 INFO:tasks.ceph_client:Setting up client nodes... 2021-03-24T20:44:34.920 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c1.client.0.keyring && sudo chmod 0644 /etc/ceph/c1.client.0.keyring 2021-03-24T20:44:34.994 INFO:teuthology.orchestra.run.smithi052.stdout:creating /etc/ceph/c1.client.0.keyring 2021-03-24T20:44:35.006 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-03-24T20:44:35.007 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}} 2021-03-24T20:44:35.007 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/c1-0 2021-03-24T20:44:35.065 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-24T20:44:35.066 INFO:tasks.ceph:role: c1.osd.0 2021-03-24T20:44:35.066 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com 2021-03-24T20:44:35.066 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-03-24T20:44:35.146 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-03-24T20:44:35.147 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-24T20:44:35.147 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-24T20:44:35.147 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-24T20:44:35.147 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2021-03-24T20:44:35.148 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-24T20:44:35.148 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-24T20:44:35.148 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-24T20:44:35.149 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-24T20:44:35.168 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2021-03-24T20:44:35.169 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c1-0 2021-03-24T20:44:35.230 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-0 2021-03-24T20:44:35.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T20:44:35.285 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: /sbin/restorecon: command not found 2021-03-24T20:44:35.286 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/c1-1 2021-03-24T20:44:35.345 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-24T20:44:35.347 INFO:tasks.ceph:role: c1.osd.1 2021-03-24T20:44:35.347 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com 2021-03-24T20:44:35.348 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-03-24T20:44:35.431 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-03-24T20:44:35.431 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-24T20:44:35.431 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-24T20:44:35.432 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-24T20:44:35.432 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2021-03-24T20:44:35.432 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-24T20:44:35.432 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-24T20:44:35.432 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-24T20:44:35.433 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-24T20:44:35.434 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2021-03-24T20:44:35.434 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c1-1 2021-03-24T20:44:35.503 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-1 2021-03-24T20:44:35.557 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: /sbin/restorecon: command not found 2021-03-24T20:44:35.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T20:44:35.559 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/osd/c1-2 2021-03-24T20:44:35.618 INFO:tasks.ceph:roles_to_devs: {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-24T20:44:35.618 INFO:tasks.ceph:role: c1.osd.2 2021-03-24T20:44:35.619 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com 2021-03-24T20:44:35.619 DEBUG:teuthology.orchestra.run.smithi052:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-03-24T20:44:35.702 INFO:teuthology.orchestra.run.smithi052.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-03-24T20:44:35.703 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-24T20:44:35.703 INFO:teuthology.orchestra.run.smithi052.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-24T20:44:35.703 INFO:teuthology.orchestra.run.smithi052.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-24T20:44:35.703 INFO:teuthology.orchestra.run.smithi052.stdout: = sunit=0 swidth=0 blks 2021-03-24T20:44:35.704 INFO:teuthology.orchestra.run.smithi052.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-24T20:44:35.704 INFO:teuthology.orchestra.run.smithi052.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-24T20:44:35.704 INFO:teuthology.orchestra.run.smithi052.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-24T20:44:35.704 INFO:teuthology.orchestra.run.smithi052.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-24T20:44:35.706 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi052.front.sepia.ceph.com -o noatime 2021-03-24T20:44:35.706 DEBUG:teuthology.orchestra.run.smithi052:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c1-2 2021-03-24T20:44:35.772 DEBUG:teuthology.orchestra.run.smithi052:> sudo /sbin/restorecon /var/lib/ceph/osd/c1-2 2021-03-24T20:44:35.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T20:44:35.831 INFO:teuthology.orchestra.run.smithi052.stderr:sudo: /sbin/restorecon: command not found 2021-03-24T20:44:35.832 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c1.monmap 2021-03-24T20:44:35.983 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:35.984+0000 7f6b94d2bd80 -1 auth: error reading file: /var/lib/ceph/osd/c1-0/keyring: can't open /var/lib/ceph/osd/c1-0/keyring: (2) No such file or directory 2021-03-24T20:44:35.984 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:35.984+0000 7f6b94d2bd80 -1 created new key in keyring /var/lib/ceph/osd/c1-0/keyring 2021-03-24T20:44:35.984 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:35.984+0000 7f6b94d2bd80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-03-24T20:44:35.984 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:35.984+0000 7f6b94d2bd80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-03-24T20:44:35.984 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:35.984+0000 7f6b94d2bd80 -1 bluestore(/var/lib/ceph/osd/c1-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-0/block: (2) No such file or directory 2021-03-24T20:44:35.985 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:35.984+0000 7f6b94d2bd80 -1 bluestore(/var/lib/ceph/osd/c1-0) _read_fsid unparsable uuid 2021-03-24T20:44:35.995 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:35.996+0000 7f6b94d2bd80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-24T20:44:38.134 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-0 2021-03-24T20:44:38.150 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c1.monmap 2021-03-24T20:44:38.232 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:38.232+0000 7f3f5c008d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-1/keyring: can't open /var/lib/ceph/osd/c1-1/keyring: (2) No such file or directory 2021-03-24T20:44:38.232 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:38.232+0000 7f3f5c008d80 -1 created new key in keyring /var/lib/ceph/osd/c1-1/keyring 2021-03-24T20:44:38.232 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:38.232+0000 7f3f5c008d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-03-24T20:44:38.233 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:38.232+0000 7f3f5c008d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-03-24T20:44:38.233 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:38.232+0000 7f3f5c008d80 -1 bluestore(/var/lib/ceph/osd/c1-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-1/block: (2) No such file or directory 2021-03-24T20:44:38.233 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:38.232+0000 7f3f5c008d80 -1 bluestore(/var/lib/ceph/osd/c1-1) _read_fsid unparsable uuid 2021-03-24T20:44:38.243 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:38.244+0000 7f3f5c008d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-24T20:44:40.141 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-1 2021-03-24T20:44:40.206 DEBUG:teuthology.orchestra.run.smithi052:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c1.monmap 2021-03-24T20:44:40.288 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:40.288+0000 7f0ce36c0d80 -1 auth: error reading file: /var/lib/ceph/osd/c1-2/keyring: can't open /var/lib/ceph/osd/c1-2/keyring: (2) No such file or directory 2021-03-24T20:44:40.288 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:40.288+0000 7f0ce36c0d80 -1 created new key in keyring /var/lib/ceph/osd/c1-2/keyring 2021-03-24T20:44:40.288 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:40.288+0000 7f0ce36c0d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-03-24T20:44:40.289 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:40.288+0000 7f0ce36c0d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-03-24T20:44:40.289 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:40.288+0000 7f0ce36c0d80 -1 bluestore(/var/lib/ceph/osd/c1-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c1-2/block: (2) No such file or directory 2021-03-24T20:44:40.289 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:40.288+0000 7f0ce36c0d80 -1 bluestore(/var/lib/ceph/osd/c1-2) _read_fsid unparsable uuid 2021-03-24T20:44:40.299 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:44:40.300+0000 7f0ce36c0d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-24T20:44:42.188 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c1-2 2021-03-24T20:44:42.245 INFO:tasks.ceph:Reading keys from all nodes... 2021-03-24T20:44:42.246 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:44:42.246 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/mgr/c1-x/keyring of=/dev/stdout 2021-03-24T20:44:42.305 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:44:42.306 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/c1-0/keyring of=/dev/stdout 2021-03-24T20:44:42.360 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:44:42.361 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/c1-1/keyring of=/dev/stdout 2021-03-24T20:44:42.417 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:44:42.418 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/c1-2/keyring of=/dev/stdout 2021-03-24T20:44:42.473 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:44:42.473 DEBUG:teuthology.orchestra.run.smithi052:> dd if=/etc/ceph/c1.client.0.keyring of=/dev/stdout 2021-03-24T20:44:42.520 INFO:tasks.ceph:Adding keys to all mons... 2021-03-24T20:44:42.520 DEBUG:teuthology.orchestra.run.smithi052:> sudo tee -a /etc/ceph/c1.keyring 2021-03-24T20:44:42.575 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr.x] 2021-03-24T20:44:42.575 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQCypFtgxaynNhAAMJZetKs1qklptoQC4859TQ== 2021-03-24T20:44:42.576 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.0] 2021-03-24T20:44:42.576 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQCzpFtg09GxOhAA+tQ9Vzm62aj2AKUqXZD1gA== 2021-03-24T20:44:42.576 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.1] 2021-03-24T20:44:42.576 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC2pFtgN2vuDRAAiS3Cki3ei3VuVEpcgV2tIA== 2021-03-24T20:44:42.577 INFO:teuthology.orchestra.run.smithi052.stdout:[osd.2] 2021-03-24T20:44:42.577 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQC4pFtg/HdDERAAW7On/Rd30fh/AYTqZE1PUQ== 2021-03-24T20:44:42.577 INFO:teuthology.orchestra.run.smithi052.stdout:[client.0] 2021-03-24T20:44:42.577 INFO:teuthology.orchestra.run.smithi052.stdout: key = AQCypFtgDdBcOxAApxcsAxElD/odB7SFcLY4NQ== 2021-03-24T20:44:42.579 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-24T20:44:42.665 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-24T20:44:42.748 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-24T20:44:42.831 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-24T20:44:42.915 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-24T20:44:43.003 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-03-24T20:44:43.004 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /var/lib/ceph/mon/c1-a 2021-03-24T20:44:43.019 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c1 --mkfs -i a --monmap /home/ubuntu/cephtest/c1.monmap --keyring /etc/ceph/c1.keyring 2021-03-24T20:44:43.122 DEBUG:teuthology.orchestra.run.smithi052:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c1-a 2021-03-24T20:44:43.137 DEBUG:teuthology.orchestra.run.smithi052:> rm -- /home/ubuntu/cephtest/c1.monmap 2021-03-24T20:44:43.184 INFO:tasks.ceph:Starting mon daemons in cluster c1... 2021-03-24T20:44:43.185 INFO:tasks.ceph.c1.mon.a:Restarting daemon 2021-03-24T20:44:43.186 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c1 -i a 2021-03-24T20:44:43.228 INFO:tasks.ceph.c1.mon.a:Started 2021-03-24T20:44:43.229 INFO:tasks.ceph:Starting mgr daemons in cluster c1... 2021-03-24T20:44:43.230 INFO:tasks.ceph.c1.mgr.x:Restarting daemon 2021-03-24T20:44:43.230 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c1 -i x 2021-03-24T20:44:43.234 INFO:tasks.ceph.c1.mgr.x:Started 2021-03-24T20:44:43.235 INFO:tasks.ceph:Setting crush tunables to default 2021-03-24T20:44:43.236 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c1 osd crush tunables default 2021-03-24T20:44:43.670 INFO:teuthology.orchestra.run.smithi052.stderr:adjusted tunables profile to default 2021-03-24T20:44:43.687 INFO:tasks.ceph:Starting osd daemons in cluster c1... 2021-03-24T20:44:43.687 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:44:43.688 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/c1-0/fsid of=/dev/stdout 2021-03-24T20:44:43.702 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:44:43.702 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/c1-1/fsid of=/dev/stdout 2021-03-24T20:44:43.758 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:44:43.759 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd if=/var/lib/ceph/osd/c1-2/fsid of=/dev/stdout 2021-03-24T20:44:43.815 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c1 osd new c13babb7-2f09-4437-9256-5b6291c9e646 0 2021-03-24T20:44:44.064 INFO:teuthology.orchestra.run.smithi052.stdout:0 2021-03-24T20:44:44.081 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c1 osd new eafd947e-a358-4639-90ca-82fa477f5c89 1 2021-03-24T20:44:44.331 INFO:teuthology.orchestra.run.smithi052.stdout:1 2021-03-24T20:44:44.343 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c1 osd new 0ccf2543-f2e4-452e-ae1e-743c78a54921 2 2021-03-24T20:44:44.540 INFO:teuthology.orchestra.run.smithi052.stdout:2 2021-03-24T20:44:44.553 INFO:tasks.ceph.c1.osd.0:Restarting daemon 2021-03-24T20:44:44.554 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 0 2021-03-24T20:44:44.575 INFO:tasks.ceph.c1.osd.0:Started 2021-03-24T20:44:44.576 INFO:tasks.ceph.c1.osd.1:Restarting daemon 2021-03-24T20:44:44.576 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 1 2021-03-24T20:44:44.594 INFO:tasks.ceph.c1.osd.1:Started 2021-03-24T20:44:44.595 INFO:tasks.ceph.c1.osd.2:Restarting daemon 2021-03-24T20:44:44.595 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c1 -i 2 2021-03-24T20:44:44.614 INFO:tasks.ceph.c1.osd.2:Started 2021-03-24T20:44:44.614 INFO:tasks.ceph:Waiting for OSDs to come up 2021-03-24T20:44:44.916 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-03-24T20:44:45.098 INFO:teuthology.misc.health.smithi052.stdout: 2021-03-24T20:44:45.098 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":5,"fsid":"91c6d487-aa59-4b0e-bf1a-56b2e36b57d7","created":"2021-03-24T20:44:43.350088+0000","modified":"2021-03-24T20:44:44.538345+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"c13babb7-2f09-4437-9256-5b6291c9e646","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"eafd947e-a358-4639-90ca-82fa477f5c89","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"0ccf2543-f2e4-452e-ae1e-743c78a54921","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-24T20:44:45.114 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-03-24T20:44:45.187 INFO:tasks.ceph.c1.osd.0.smithi052.stderr:2021-03-24T20:44:45.188+0000 7f85e464dd80 -1 Falling back to public interface 2021-03-24T20:44:45.203 INFO:tasks.ceph.c1.osd.1.smithi052.stderr:2021-03-24T20:44:45.204+0000 7f575c92fd80 -1 Falling back to public interface 2021-03-24T20:44:45.215 INFO:tasks.ceph.c1.osd.2.smithi052.stderr:2021-03-24T20:44:45.216+0000 7f3596592d80 -1 Falling back to public interface 2021-03-24T20:44:45.797 INFO:tasks.ceph.c1.osd.0.smithi052.stderr:2021-03-24T20:44:45.796+0000 7f85e464dd80 -1 osd.0 0 log_to_monitors {default=true} 2021-03-24T20:44:45.809 INFO:tasks.ceph.c1.osd.1.smithi052.stderr:2021-03-24T20:44:45.808+0000 7f575c92fd80 -1 osd.1 0 log_to_monitors {default=true} 2021-03-24T20:44:45.832 INFO:tasks.ceph.c1.osd.2.smithi052.stderr:2021-03-24T20:44:45.832+0000 7f3596592d80 -1 osd.2 0 log_to_monitors {default=true} 2021-03-24T20:44:47.407 INFO:tasks.ceph.c1.osd.0.smithi052.stderr:2021-03-24T20:44:47.408+0000 7f85ce099700 -1 osd.0 0 waiting for initial osdmap 2021-03-24T20:44:47.407 INFO:tasks.ceph.c1.osd.2.smithi052.stderr:2021-03-24T20:44:47.408+0000 7f357ffde700 -1 osd.2 0 waiting for initial osdmap 2021-03-24T20:44:47.408 INFO:tasks.ceph.c1.osd.1.smithi052.stderr:2021-03-24T20:44:47.408+0000 7f574637b700 -1 osd.1 0 waiting for initial osdmap 2021-03-24T20:44:51.418 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c1 osd dump --format=json 2021-03-24T20:44:51.689 INFO:teuthology.misc.health.smithi052.stdout: 2021-03-24T20:44:51.689 INFO:teuthology.misc.health.smithi052.stdout:{"epoch":10,"fsid":"91c6d487-aa59-4b0e-bf1a-56b2e36b57d7","created":"2021-03-24T20:44:43.350088+0000","modified":"2021-03-24T20:44:50.419493+0000","last_up_change":"2021-03-24T20:44:48.410807+0000","last_in_change":"2021-03-24T20:44:48.410807+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-24T20:44:47.385777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"c13babb7-2f09-4437-9256-5b6291c9e646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":12674},{"type":"v1","addr":"172.21.15.52:6801","nonce":12674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":12674},{"type":"v1","addr":"172.21.15.52:6803","nonce":12674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":12674},{"type":"v1","addr":"172.21.15.52:6807","nonce":12674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":12674},{"type":"v1","addr":"172.21.15.52:6805","nonce":12674}]},"public_addr":"172.21.15.52:6801/12674","cluster_addr":"172.21.15.52:6803/12674","heartbeat_back_addr":"172.21.15.52:6807/12674","heartbeat_front_addr":"172.21.15.52:6805/12674","state":["exists","up"]},{"osd":1,"uuid":"eafd947e-a358-4639-90ca-82fa477f5c89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":8,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":12675},{"type":"v1","addr":"172.21.15.52:6809","nonce":12675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":12675},{"type":"v1","addr":"172.21.15.52:6811","nonce":12675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":12675},{"type":"v1","addr":"172.21.15.52:6815","nonce":12675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":12675},{"type":"v1","addr":"172.21.15.52:6813","nonce":12675}]},"public_addr":"172.21.15.52:6809/12675","cluster_addr":"172.21.15.52:6811/12675","heartbeat_back_addr":"172.21.15.52:6815/12675","heartbeat_front_addr":"172.21.15.52:6813/12675","state":["exists","up"]},{"osd":2,"uuid":"0ccf2543-f2e4-452e-ae1e-743c78a54921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":12698},{"type":"v1","addr":"172.21.15.52:6817","nonce":12698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":12698},{"type":"v1","addr":"172.21.15.52:6819","nonce":12698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":12698},{"type":"v1","addr":"172.21.15.52:6823","nonce":12698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":12698},{"type":"v1","addr":"172.21.15.52:6821","nonce":12698}]},"public_addr":"172.21.15.52:6817/12698","cluster_addr":"172.21.15.52:6819/12698","heartbeat_back_addr":"172.21.15.52:6823/12698","heartbeat_front_addr":"172.21.15.52:6821/12698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:44:46.830510+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:44:46.766548+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:44:46.875880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-24T20:44:51.701 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-03-24T20:44:51.702 INFO:tasks.ceph:Creating RBD pool 2021-03-24T20:44:51.702 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c1 osd pool create rbd 8 2021-03-24T20:44:52.447 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'rbd' created 2021-03-24T20:44:52.460 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c1 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-03-24T20:44:53.457 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rbd' on pool 'rbd' 2021-03-24T20:44:53.469 INFO:tasks.ceph:Starting mds daemons in cluster c1... 2021-03-24T20:44:53.470 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-03-24T20:44:53.471 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-03-24T20:44:53.746 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:53.747 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":12,"fsid":"91c6d487-aa59-4b0e-bf1a-56b2e36b57d7","created":"2021-03-24T20:44:43.350088+0000","modified":"2021-03-24T20:44:53.437299+0000","last_up_change":"2021-03-24T20:44:48.410807+0000","last_in_change":"2021-03-24T20:44:48.410807+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-24T20:44:47.385777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-24T20:44:52.038659+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"12","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c13babb7-2f09-4437-9256-5b6291c9e646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":12674},{"type":"v1","addr":"172.21.15.52:6801","nonce":12674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":12674},{"type":"v1","addr":"172.21.15.52:6803","nonce":12674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":12674},{"type":"v1","addr":"172.21.15.52:6807","nonce":12674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":12674},{"type":"v1","addr":"172.21.15.52:6805","nonce":12674}]},"public_addr":"172.21.15.52:6801/12674","cluster_addr":"172.21.15.52:6803/12674","heartbeat_back_addr":"172.21.15.52:6807/12674","heartbeat_front_addr":"172.21.15.52:6805/12674","state":["exists","up"]},{"osd":1,"uuid":"eafd947e-a358-4639-90ca-82fa477f5c89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":12675},{"type":"v1","addr":"172.21.15.52:6809","nonce":12675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":12675},{"type":"v1","addr":"172.21.15.52:6811","nonce":12675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":12675},{"type":"v1","addr":"172.21.15.52:6815","nonce":12675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":12675},{"type":"v1","addr":"172.21.15.52:6813","nonce":12675}]},"public_addr":"172.21.15.52:6809/12675","cluster_addr":"172.21.15.52:6811/12675","heartbeat_back_addr":"172.21.15.52:6815/12675","heartbeat_front_addr":"172.21.15.52:6813/12675","state":["exists","up"]},{"osd":2,"uuid":"0ccf2543-f2e4-452e-ae1e-743c78a54921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":12698},{"type":"v1","addr":"172.21.15.52:6817","nonce":12698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":12698},{"type":"v1","addr":"172.21.15.52:6819","nonce":12698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":12698},{"type":"v1","addr":"172.21.15.52:6823","nonce":12698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":12698},{"type":"v1","addr":"172.21.15.52:6821","nonce":12698}]},"public_addr":"172.21.15.52:6817/12698","cluster_addr":"172.21.15.52:6819/12698","heartbeat_back_addr":"172.21.15.52:6823/12698","heartbeat_front_addr":"172.21.15.52:6821/12698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:44:46.830510+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:44:46.766548+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:44:46.875880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-24T20:44:53.760 INFO:tasks.ceph.ceph_manager.c1:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-24T20:44:47.385777+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-03-24T20:44:52.038659+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '12', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-03-24T20:44:53.761 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get device_health_metrics pg_num 2021-03-24T20:44:54.048 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 1 2021-03-24T20:44:54.069 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd pool get rbd pg_num 2021-03-24T20:44:54.348 INFO:teuthology.orchestra.run.smithi052.stdout:pg_num: 8 2021-03-24T20:44:54.362 INFO:tasks.ceph:Waiting until c1 daemons up and pgs clean... 2021-03-24T20:44:54.363 INFO:tasks.ceph.ceph_manager.c1:waiting for mgr available 2021-03-24T20:44:54.363 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 mgr dump --format=json 2021-03-24T20:44:54.674 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:54.692 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":4,"active_gid":4098,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6824","nonce":12483},{"type":"v1","addr":"172.21.15.52:6825","nonce":12483}]},"active_addr":"172.21.15.52:6825/12483","active_change":"2021-03-24T20:44:47.294059+0000","active_mgr_features":4540138292840890367,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":1995448422}]},{"addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":2509174012}]},{"addrvec":[{"type":"v2","addr":"172.21.15.52:0","nonce":1715423268}]}]}} 2021-03-24T20:44:54.700 INFO:tasks.ceph.ceph_manager.c1:mgr available! 2021-03-24T20:44:54.701 INFO:tasks.ceph.ceph_manager.c1:waiting for all up 2021-03-24T20:44:54.701 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-03-24T20:44:54.978 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:54.978 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":13,"fsid":"91c6d487-aa59-4b0e-bf1a-56b2e36b57d7","created":"2021-03-24T20:44:43.350088+0000","modified":"2021-03-24T20:44:54.442473+0000","last_up_change":"2021-03-24T20:44:48.410807+0000","last_in_change":"2021-03-24T20:44:48.410807+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-24T20:44:47.385777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-24T20:44:52.038659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c13babb7-2f09-4437-9256-5b6291c9e646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":12674},{"type":"v1","addr":"172.21.15.52:6801","nonce":12674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":12674},{"type":"v1","addr":"172.21.15.52:6803","nonce":12674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":12674},{"type":"v1","addr":"172.21.15.52:6807","nonce":12674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":12674},{"type":"v1","addr":"172.21.15.52:6805","nonce":12674}]},"public_addr":"172.21.15.52:6801/12674","cluster_addr":"172.21.15.52:6803/12674","heartbeat_back_addr":"172.21.15.52:6807/12674","heartbeat_front_addr":"172.21.15.52:6805/12674","state":["exists","up"]},{"osd":1,"uuid":"eafd947e-a358-4639-90ca-82fa477f5c89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":12675},{"type":"v1","addr":"172.21.15.52:6809","nonce":12675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":12675},{"type":"v1","addr":"172.21.15.52:6811","nonce":12675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":12675},{"type":"v1","addr":"172.21.15.52:6815","nonce":12675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":12675},{"type":"v1","addr":"172.21.15.52:6813","nonce":12675}]},"public_addr":"172.21.15.52:6809/12675","cluster_addr":"172.21.15.52:6811/12675","heartbeat_back_addr":"172.21.15.52:6815/12675","heartbeat_front_addr":"172.21.15.52:6813/12675","state":["exists","up"]},{"osd":2,"uuid":"0ccf2543-f2e4-452e-ae1e-743c78a54921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":12698},{"type":"v1","addr":"172.21.15.52:6817","nonce":12698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":12698},{"type":"v1","addr":"172.21.15.52:6819","nonce":12698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":12698},{"type":"v1","addr":"172.21.15.52:6823","nonce":12698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":12698},{"type":"v1","addr":"172.21.15.52:6821","nonce":12698}]},"public_addr":"172.21.15.52:6817/12698","cluster_addr":"172.21.15.52:6819/12698","heartbeat_back_addr":"172.21.15.52:6823/12698","heartbeat_front_addr":"172.21.15.52:6821/12698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:44:46.830510+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:44:46.766548+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:44:46.875880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-24T20:44:54.992 INFO:tasks.ceph.ceph_manager.c1:all up! 2021-03-24T20:44:54.993 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-03-24T20:44:55.274 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:55.275 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":13,"fsid":"91c6d487-aa59-4b0e-bf1a-56b2e36b57d7","created":"2021-03-24T20:44:43.350088+0000","modified":"2021-03-24T20:44:54.442473+0000","last_up_change":"2021-03-24T20:44:48.410807+0000","last_in_change":"2021-03-24T20:44:48.410807+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-24T20:44:47.385777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-24T20:44:52.038659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c13babb7-2f09-4437-9256-5b6291c9e646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":12674},{"type":"v1","addr":"172.21.15.52:6801","nonce":12674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":12674},{"type":"v1","addr":"172.21.15.52:6803","nonce":12674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":12674},{"type":"v1","addr":"172.21.15.52:6807","nonce":12674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":12674},{"type":"v1","addr":"172.21.15.52:6805","nonce":12674}]},"public_addr":"172.21.15.52:6801/12674","cluster_addr":"172.21.15.52:6803/12674","heartbeat_back_addr":"172.21.15.52:6807/12674","heartbeat_front_addr":"172.21.15.52:6805/12674","state":["exists","up"]},{"osd":1,"uuid":"eafd947e-a358-4639-90ca-82fa477f5c89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":12675},{"type":"v1","addr":"172.21.15.52:6809","nonce":12675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":12675},{"type":"v1","addr":"172.21.15.52:6811","nonce":12675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":12675},{"type":"v1","addr":"172.21.15.52:6815","nonce":12675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":12675},{"type":"v1","addr":"172.21.15.52:6813","nonce":12675}]},"public_addr":"172.21.15.52:6809/12675","cluster_addr":"172.21.15.52:6811/12675","heartbeat_back_addr":"172.21.15.52:6815/12675","heartbeat_front_addr":"172.21.15.52:6813/12675","state":["exists","up"]},{"osd":2,"uuid":"0ccf2543-f2e4-452e-ae1e-743c78a54921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":11,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":12698},{"type":"v1","addr":"172.21.15.52:6817","nonce":12698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":12698},{"type":"v1","addr":"172.21.15.52:6819","nonce":12698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":12698},{"type":"v1","addr":"172.21.15.52:6823","nonce":12698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":12698},{"type":"v1","addr":"172.21.15.52:6821","nonce":12698}]},"public_addr":"172.21.15.52:6817/12698","cluster_addr":"172.21.15.52:6819/12698","heartbeat_back_addr":"172.21.15.52:6823/12698","heartbeat_front_addr":"172.21.15.52:6821/12698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:44:46.830510+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:44:46.766548+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:44:46.875880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-24T20:44:55.289 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.0 flush_pg_stats 2021-03-24T20:44:55.450 INFO:teuthology.orchestra.run.smithi052.stdout:34359738371 2021-03-24T20:44:55.451 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.1 flush_pg_stats 2021-03-24T20:44:55.616 INFO:teuthology.orchestra.run.smithi052.stdout:34359738371 2021-03-24T20:44:55.616 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 tell osd.2 flush_pg_stats 2021-03-24T20:44:55.773 INFO:teuthology.orchestra.run.smithi052.stdout:34359738372 2021-03-24T20:44:55.773 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-03-24T20:44:56.051 INFO:teuthology.orchestra.run.smithi052.stdout:34359738370 2021-03-24T20:44:56.071 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2021-03-24T20:44:57.072 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-03-24T20:44:57.354 INFO:teuthology.orchestra.run.smithi052.stdout:34359738370 2021-03-24T20:44:57.368 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738370 for osd.0 2021-03-24T20:44:58.369 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.0 2021-03-24T20:44:58.655 INFO:teuthology.orchestra.run.smithi052.stdout:34359738371 2021-03-24T20:44:58.670 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.0 2021-03-24T20:44:58.670 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.1 2021-03-24T20:44:58.941 INFO:teuthology.orchestra.run.smithi052.stdout:34359738371 2021-03-24T20:44:58.955 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738371 got 34359738371 for osd.1 2021-03-24T20:44:58.956 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd last-stat-seq osd.2 2021-03-24T20:44:59.231 INFO:teuthology.orchestra.run.smithi052.stdout:34359738372 2021-03-24T20:44:59.250 INFO:tasks.ceph.ceph_manager.c1:need seq 34359738372 got 34359738372 for osd.2 2021-03-24T20:44:59.250 INFO:tasks.ceph.ceph_manager.c1:waiting for clean 2021-03-24T20:44:59.250 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-03-24T20:44:59.522 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:59.524 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2021-03-24T20:44:59.536 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-03-24T20:44:59.305469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146080,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426720,"statfs":{"total":322122547200,"available":318900961280,"internally_reserved":3221225472,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.860588"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.459912+0000","last_change":"2021-03-24T20:44:53.457418+0000","last_active":"2021-03-24T20:44:53.459912+0000","last_peered":"2021-03-24T20:44:53.459912+0000","last_clean":"2021-03-24T20:44:53.459912+0000","last_became_active":"2021-03-24T20:44:53.456962+0000","last_became_peered":"2021-03-24T20:44:53.456962+0000","last_unstale":"2021-03-24T20:44:53.459912+0000","last_undegraded":"2021-03-24T20:44:53.459912+0000","last_fullsized":"2021-03-24T20:44:53.459912+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_clean_scrub_stamp":"2021-03-24T20:44:52.430525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.459883+0000","last_change":"2021-03-24T20:44:53.457423+0000","last_active":"2021-03-24T20:44:53.459883+0000","last_peered":"2021-03-24T20:44:53.459883+0000","last_clean":"2021-03-24T20:44:53.459883+0000","last_became_active":"2021-03-24T20:44:53.457030+0000","last_became_peered":"2021-03-24T20:44:53.457030+0000","last_unstale":"2021-03-24T20:44:53.459883+0000","last_undegraded":"2021-03-24T20:44:53.459883+0000","last_fullsized":"2021-03-24T20:44:53.459883+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_clean_scrub_stamp":"2021-03-24T20:44:52.430525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.460091+0000","last_change":"2021-03-24T20:44:53.458131+0000","last_active":"2021-03-24T20:44:53.460091+0000","last_peered":"2021-03-24T20:44:53.460091+0000","last_clean":"2021-03-24T20:44:53.460091+0000","last_became_active":"2021-03-24T20:44:53.457730+0000","last_became_peered":"2021-03-24T20:44:53.457730+0000","last_unstale":"2021-03-24T20:44:53.460091+0000","last_undegraded":"2021-03-24T20:44:53.460091+0000","last_fullsized":"2021-03-24T20:44:53.460091+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_clean_scrub_stamp":"2021-03-24T20:44:52.430525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.459223+0000","last_change":"2021-03-24T20:44:53.458272+0000","last_active":"2021-03-24T20:44:53.459223+0000","last_peered":"2021-03-24T20:44:53.459223+0000","last_clean":"2021-03-24T20:44:53.459223+0000","last_became_active":"2021-03-24T20:44:53.456501+0000","last_became_peered":"2021-03-24T20:44:53.456501+0000","last_unstale":"2021-03-24T20:44:53.459223+0000","last_undegraded":"2021-03-24T20:44:53.459223+0000","last_fullsized":"2021-03-24T20:44:53.459223+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_clean_scrub_stamp":"2021-03-24T20:44:52.430525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.453628+0000","last_change":"2021-03-24T20:44:49.708529+0000","last_active":"2021-03-24T20:44:53.453628+0000","last_peered":"2021-03-24T20:44:53.453628+0000","last_clean":"2021-03-24T20:44:53.453628+0000","last_became_active":"2021-03-24T20:44:49.707387+0000","last_became_peered":"2021-03-24T20:44:49.707387+0000","last_unstale":"2021-03-24T20:44:53.453628+0000","last_undegraded":"2021-03-24T20:44:53.453628+0000","last_fullsized":"2021-03-24T20:44:53.453628+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:47.397878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:47.397878+0000","last_clean_scrub_stamp":"2021-03-24T20:44:47.397878+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.459576+0000","last_change":"2021-03-24T20:44:53.457144+0000","last_active":"2021-03-24T20:44:53.459576+0000","last_peered":"2021-03-24T20:44:53.459576+0000","last_clean":"2021-03-24T20:44:53.459576+0000","last_became_active":"2021-03-24T20:44:53.456155+0000","last_became_peered":"2021-03-24T20:44:53.456155+0000","last_unstale":"2021-03-24T20:44:53.459576+0000","last_undegraded":"2021-03-24T20:44:53.459576+0000","last_fullsized":"2021-03-24T20:44:53.459576+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_clean_scrub_stamp":"2021-03-24T20:44:52.430525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.459579+0000","last_change":"2021-03-24T20:44:53.458389+0000","last_active":"2021-03-24T20:44:53.459579+0000","last_peered":"2021-03-24T20:44:53.459579+0000","last_clean":"2021-03-24T20:44:53.459579+0000","last_became_active":"2021-03-24T20:44:53.457538+0000","last_became_peered":"2021-03-24T20:44:53.457538+0000","last_unstale":"2021-03-24T20:44:53.459579+0000","last_undegraded":"2021-03-24T20:44:53.459579+0000","last_fullsized":"2021-03-24T20:44:53.459579+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_clean_scrub_stamp":"2021-03-24T20:44:52.430525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.458175+0000","last_change":"2021-03-24T20:44:53.457507+0000","last_active":"2021-03-24T20:44:53.458175+0000","last_peered":"2021-03-24T20:44:53.458175+0000","last_clean":"2021-03-24T20:44:53.458175+0000","last_became_active":"2021-03-24T20:44:53.457088+0000","last_became_peered":"2021-03-24T20:44:53.457088+0000","last_unstale":"2021-03-24T20:44:53.458175+0000","last_undegraded":"2021-03-24T20:44:53.458175+0000","last_fullsized":"2021-03-24T20:44:53.458175+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_clean_scrub_stamp":"2021-03-24T20:44:52.430525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.460085+0000","last_change":"2021-03-24T20:44:53.457599+0000","last_active":"2021-03-24T20:44:53.460085+0000","last_peered":"2021-03-24T20:44:53.460085+0000","last_clean":"2021-03-24T20:44:53.460085+0000","last_became_active":"2021-03-24T20:44:53.456657+0000","last_became_peered":"2021-03-24T20:44:53.456657+0000","last_unstale":"2021-03-24T20:44:53.460085+0000","last_undegraded":"2021-03-24T20:44:53.460085+0000","last_fullsized":"2021-03-24T20:44:53.460085+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_clean_scrub_stamp":"2021-03-24T20:44:52.430525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]}]}} 2021-03-24T20:44:59.537 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 pg dump --format=json 2021-03-24T20:44:59.810 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:44:59.812 INFO:teuthology.orchestra.run.smithi052.stderr:dumped all 2021-03-24T20:44:59.824 INFO:teuthology.orchestra.run.smithi052.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2021-03-24T20:44:59.305469+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":15,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146080,"kb_used_data":328,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426720,"statfs":{"total":322122547200,"available":318900961280,"internally_reserved":3221225472,"allocated":335872,"data_stored":95895,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.860588"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.459912+0000","last_change":"2021-03-24T20:44:53.457418+0000","last_active":"2021-03-24T20:44:53.459912+0000","last_peered":"2021-03-24T20:44:53.459912+0000","last_clean":"2021-03-24T20:44:53.459912+0000","last_became_active":"2021-03-24T20:44:53.456962+0000","last_became_peered":"2021-03-24T20:44:53.456962+0000","last_unstale":"2021-03-24T20:44:53.459912+0000","last_undegraded":"2021-03-24T20:44:53.459912+0000","last_fullsized":"2021-03-24T20:44:53.459912+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_clean_scrub_stamp":"2021-03-24T20:44:52.430525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.459883+0000","last_change":"2021-03-24T20:44:53.457423+0000","last_active":"2021-03-24T20:44:53.459883+0000","last_peered":"2021-03-24T20:44:53.459883+0000","last_clean":"2021-03-24T20:44:53.459883+0000","last_became_active":"2021-03-24T20:44:53.457030+0000","last_became_peered":"2021-03-24T20:44:53.457030+0000","last_unstale":"2021-03-24T20:44:53.459883+0000","last_undegraded":"2021-03-24T20:44:53.459883+0000","last_fullsized":"2021-03-24T20:44:53.459883+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_clean_scrub_stamp":"2021-03-24T20:44:52.430525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.460091+0000","last_change":"2021-03-24T20:44:53.458131+0000","last_active":"2021-03-24T20:44:53.460091+0000","last_peered":"2021-03-24T20:44:53.460091+0000","last_clean":"2021-03-24T20:44:53.460091+0000","last_became_active":"2021-03-24T20:44:53.457730+0000","last_became_peered":"2021-03-24T20:44:53.457730+0000","last_unstale":"2021-03-24T20:44:53.460091+0000","last_undegraded":"2021-03-24T20:44:53.460091+0000","last_fullsized":"2021-03-24T20:44:53.460091+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_clean_scrub_stamp":"2021-03-24T20:44:52.430525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.459223+0000","last_change":"2021-03-24T20:44:53.458272+0000","last_active":"2021-03-24T20:44:53.459223+0000","last_peered":"2021-03-24T20:44:53.459223+0000","last_clean":"2021-03-24T20:44:53.459223+0000","last_became_active":"2021-03-24T20:44:53.456501+0000","last_became_peered":"2021-03-24T20:44:53.456501+0000","last_unstale":"2021-03-24T20:44:53.459223+0000","last_undegraded":"2021-03-24T20:44:53.459223+0000","last_fullsized":"2021-03-24T20:44:53.459223+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_clean_scrub_stamp":"2021-03-24T20:44:52.430525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"9'1","reported_seq":"13","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.453628+0000","last_change":"2021-03-24T20:44:49.708529+0000","last_active":"2021-03-24T20:44:53.453628+0000","last_peered":"2021-03-24T20:44:53.453628+0000","last_clean":"2021-03-24T20:44:53.453628+0000","last_became_active":"2021-03-24T20:44:49.707387+0000","last_became_peered":"2021-03-24T20:44:49.707387+0000","last_unstale":"2021-03-24T20:44:53.453628+0000","last_undegraded":"2021-03-24T20:44:53.453628+0000","last_fullsized":"2021-03-24T20:44:53.453628+0000","mapping_epoch":8,"log_start":"0'0","ondisk_log_start":"0'0","created":7,"last_epoch_clean":9,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:47.397878+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:47.397878+0000","last_clean_scrub_stamp":"2021-03-24T20:44:47.397878+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.459576+0000","last_change":"2021-03-24T20:44:53.457144+0000","last_active":"2021-03-24T20:44:53.459576+0000","last_peered":"2021-03-24T20:44:53.459576+0000","last_clean":"2021-03-24T20:44:53.459576+0000","last_became_active":"2021-03-24T20:44:53.456155+0000","last_became_peered":"2021-03-24T20:44:53.456155+0000","last_unstale":"2021-03-24T20:44:53.459576+0000","last_undegraded":"2021-03-24T20:44:53.459576+0000","last_fullsized":"2021-03-24T20:44:53.459576+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_clean_scrub_stamp":"2021-03-24T20:44:52.430525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.459579+0000","last_change":"2021-03-24T20:44:53.458389+0000","last_active":"2021-03-24T20:44:53.459579+0000","last_peered":"2021-03-24T20:44:53.459579+0000","last_clean":"2021-03-24T20:44:53.459579+0000","last_became_active":"2021-03-24T20:44:53.457538+0000","last_became_peered":"2021-03-24T20:44:53.457538+0000","last_unstale":"2021-03-24T20:44:53.459579+0000","last_undegraded":"2021-03-24T20:44:53.459579+0000","last_fullsized":"2021-03-24T20:44:53.459579+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_clean_scrub_stamp":"2021-03-24T20:44:52.430525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.458175+0000","last_change":"2021-03-24T20:44:53.457507+0000","last_active":"2021-03-24T20:44:53.458175+0000","last_peered":"2021-03-24T20:44:53.458175+0000","last_clean":"2021-03-24T20:44:53.458175+0000","last_became_active":"2021-03-24T20:44:53.457088+0000","last_became_peered":"2021-03-24T20:44:53.457088+0000","last_unstale":"2021-03-24T20:44:53.458175+0000","last_undegraded":"2021-03-24T20:44:53.458175+0000","last_fullsized":"2021-03-24T20:44:53.458175+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_clean_scrub_stamp":"2021-03-24T20:44:52.430525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"12","state":"active+clean","last_fresh":"2021-03-24T20:44:53.460085+0000","last_change":"2021-03-24T20:44:53.457599+0000","last_active":"2021-03-24T20:44:53.460085+0000","last_peered":"2021-03-24T20:44:53.460085+0000","last_clean":"2021-03-24T20:44:53.460085+0000","last_became_active":"2021-03-24T20:44:53.456657+0000","last_became_peered":"2021-03-24T20:44:53.456657+0000","last_unstale":"2021-03-24T20:44:53.460085+0000","last_undegraded":"2021-03-24T20:44:53.460085+0000","last_fullsized":"2021-03-24T20:44:53.460085+0000","mapping_epoch":11,"log_start":"0'0","ondisk_log_start":"0'0","created":11,"last_epoch_clean":12,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:44:52.430525+0000","last_clean_scrub_stamp":"2021-03-24T20:44:52.430525+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.42199999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":0,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048688,"kb_used_data":104,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808912,"statfs":{"total":107374182400,"available":106300325888,"internally_reserved":1073741824,"allocated":106496,"data_stored":29623,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46500000000000002}]}]},{"osd":1,"up_from":8,"seq":34359738371,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":33136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]}]}} 2021-03-24T20:44:59.825 INFO:tasks.ceph.ceph_manager.c1:clean! 2021-03-24T20:44:59.826 INFO:tasks.ceph:Waiting until ceph cluster c1 is healthy... 2021-03-24T20:44:59.826 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy 2021-03-24T20:44:59.827 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 health --format=json 2021-03-24T20:45:00.152 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.166 INFO:teuthology.orchestra.run.smithi052.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-03-24T20:45:00.166 INFO:tasks.ceph.ceph_manager.c1:wait_until_healthy done 2021-03-24T20:45:00.167 INFO:teuthology.run_tasks:Running task ceph... 2021-03-24T20:45:00.179 INFO:tasks.ceph:Creating ceph cluster c2... 2021-03-24T20:45:00.179 INFO:tasks.ceph:config {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'cpu_profile': set(), 'cluster': 'c2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2021-03-24T20:45:00.180 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi/5994655', 'branch': 'wip-sage-testing-2021-03-24-1139', 'description': 'upgrade:octopus-x/rgw-multisite/{clusters frontend overrides realm tasks upgrade/secondary}', 'email': None, 'first_in_suite': False, 'job_id': '5994655', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi', 'nuke-on-error': True, 'overrides': {'admin_socket': {'branch': 'wip-sage-testing-2021-03-24-1139'}, 'ceph': {'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2', 'wait-for-scrub': False}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'branch': 'octopus', 'sha1': '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2'}}, 'rgw': {'compression type': 'random', 'frontend': 'beast'}, 'rgw-multisite': {'realm': {'is default': True, 'name': 'test-realm'}, 'zonegroups': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zonegroup', 'zones': [{'endpoints': ['c1.client.0'], 'is_default': True, 'is_master': True, 'name': 'test-zone1'}, {'endpoints': ['c2.client.0'], 'is_default': True, 'name': 'test-zone2'}, {'endpoints': ['c2.client.1'], 'is_pubsub': True, 'name': 'test-zone3'}]}]}, 'rgw-multisite-tests': {'args': ['tests.py', 'tests_ps.py'], 'branch': 'octopus', 'repo': 'https://github.com/ceph/ceph.git'}, 'workunit': {'branch': 'wip-sage-testing-2021-03-24-1139', 'sha1': '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2'}}, 'owner': 'scheduled_sage@teuthology', 'priority': 75, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['c1.mon.a', 'c1.mgr.x', 'c1.osd.0', 'c1.osd.1', 'c1.osd.2', 'c1.client.0'], ['c2.mon.a', 'c2.mgr.x', 'c2.osd.0', 'c2.osd.1', 'c2.osd.2', 'c2.client.0', 'c2.client.1']], 'sha1': '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2', 'sleep_before_teardown': 0, 'suite': 'upgrade:octopus-x', 'suite_branch': 'wip-sage-testing-2021-03-24-1139', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2', 'targets': {'smithi052.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdmeEuI/ElAp8wa3lVeLfXYahy/pX0c/EdGRCcDqpaRhyuHrjHU0PEe8ckqbf14SelozcIfJk0CStTIKUBckWCPM5LZwtXt7Vh7NQCk6AHxWQq1egfGetVeWbzuPWS+q+doQ+4EyNnnZyWjnIXPtJc4R4tr/mCnKYWRQqax2ypu7oGzF/XCnsQeiRDdY1LWNxobRIpSIEGCJfD4NgJ1MSC4TrgjAw95hrAZgI480tzXReYyKL+WsdNrH/M4fVdlNXXNn5InUbzLyLYKaUfOkbTzr0Jp6SyzBKkBzcCg7kivZwOCv/HNZuHjh3ToUWZValSR3cjBMiSadmLcUBWcQNN', 'smithi063.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJpxVWP+MN/3YlQybedqnQLiHHtDwzkWm82LhqQYZMkhGsE3UDKFv4wmJiTlzH9DcdOW55ds5k0h8KWkxBwtGUf2Rpz3o1MYiLkHk7T/pSG1CC02Iwy7Mt8Je0N3HebLVwXGmBUDu+VIuo8CZNk3dW2KYmWISpqWkywv3/IsA4aMBtRnpYGISapE0Y67MEsM7thcPSAtroKsTa4rZnYbhySBlufPLGKlHn10I53dR5brooEVN4sfHXKATbKSUmG7DxsJOWhvu3zzR+3RSZ/aS8r40WQE5wdKjMt6n6MASnDB2nX/ztpeZK1EebhPzMoK6EhSudrfaq/cOGBB2y7bTL'}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'cluster': 'c1', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2', 'wait-for-scrub': False}}, {'ceph': {'cluster': 'c2', 'conf': {'client': {'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'debug ms': 1, 'debug osd': 20}}, 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(PG_AVAILABILITY\\)'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'sha1': '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2', 'wait-for-scrub': False}}, {'parallel': ['upgrade-task']}, {'rgw': {'c1.client.0': {'port': 8000}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}}}, {'rgw-multisite': None}, {'rgw-multisite-tests': {'config': {'reconfigure_delay': 60}}}], 'teuthology_branch': 'master', 'teuthology_sha1': '6b3150e9e0aa7ca432e26f31d87920ebd77f3708', 'timestamp': '2021-03-24_19:49:41', 'tube': 'smithi', 'upgrade-task': [{'install.upgrade': {'c2.client.0': None, 'c2.client.1': None, 'c2.mgr.x': None, 'c2.mon.a': None, 'c2.osd.0': None, 'c2.osd.1': None, 'c2.osd.2': None}}], 'user': 'sage', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.9855'} 2021-03-24T20:45:00.181 DEBUG:teuthology.orchestra.run.smithi052:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-03-24T20:45:00.183 DEBUG:teuthology.orchestra.run.smithi063:> install -d -m0755 -- /home/ubuntu/cephtest/c2.data 2021-03-24T20:45:00.191 DEBUG:teuthology.orchestra.run.smithi052:> sudo install -d -m0777 -- /var/run/ceph 2021-03-24T20:45:00.231 DEBUG:teuthology.orchestra.run.smithi063:> sudo install -d -m0777 -- /var/run/ceph 2021-03-24T20:45:00.247 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:45:00.248 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/scratch_devs of=/dev/stdout 2021-03-24T20:45:00.294 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-03-24T20:45:00.295 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-03-24T20:45:00.343 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-03-24T20:45:00.343 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-24T20:45:00.343 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 6h/6d Inode: 341 Links: 1 2021-03-24T20:45:00.343 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-24T20:45:00.344 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2021-03-24 20:41:45.907450858 +0000 2021-03-24T20:45:00.344 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2021-03-24 20:41:45.903448857 +0000 2021-03-24T20:45:00.344 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2021-03-24 20:41:45.903448857 +0000 2021-03-24T20:45:00.344 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: - 2021-03-24T20:45:00.351 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2021-03-24T20:45:00.351 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2021-03-24T20:45:00.352 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.00039036 s, 1.3 MB/s 2021-03-24T20:45:00.355 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-03-24T20:45:00.403 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-03-24T20:45:00.403 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-24T20:45:00.403 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 6h/6d Inode: 348 Links: 1 2021-03-24T20:45:00.404 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-24T20:45:00.404 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2021-03-24 20:41:46.307650852 +0000 2021-03-24T20:45:00.404 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2021-03-24 20:41:46.191592855 +0000 2021-03-24T20:45:00.404 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2021-03-24 20:41:46.191592855 +0000 2021-03-24T20:45:00.405 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: - 2021-03-24T20:45:00.411 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2021-03-24T20:45:00.412 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2021-03-24T20:45:00.412 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000276117 s, 1.9 MB/s 2021-03-24T20:45:00.416 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-03-24T20:45:00.467 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-03-24T20:45:00.467 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-24T20:45:00.467 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 6h/6d Inode: 354 Links: 1 2021-03-24T20:45:00.468 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-24T20:45:00.468 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2021-03-24 20:41:46.555774851 +0000 2021-03-24T20:45:00.468 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2021-03-24 20:41:46.435714851 +0000 2021-03-24T20:45:00.468 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2021-03-24 20:41:46.435714851 +0000 2021-03-24T20:45:00.468 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: - 2021-03-24T20:45:00.477 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2021-03-24T20:45:00.477 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2021-03-24T20:45:00.478 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.00036585 s, 1.4 MB/s 2021-03-24T20:45:00.481 DEBUG:teuthology.orchestra.run.smithi063:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-03-24T20:45:00.531 INFO:teuthology.orchestra.run.smithi063.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-03-24T20:45:00.532 INFO:teuthology.orchestra.run.smithi063.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-03-24T20:45:00.532 INFO:teuthology.orchestra.run.smithi063.stdout:Device: 6h/6d Inode: 360 Links: 1 2021-03-24T20:45:00.533 INFO:teuthology.orchestra.run.smithi063.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-03-24T20:45:00.533 INFO:teuthology.orchestra.run.smithi063.stdout:Access: 2021-03-24 20:41:46.927960845 +0000 2021-03-24T20:45:00.533 INFO:teuthology.orchestra.run.smithi063.stdout:Modify: 2021-03-24 20:41:46.819906848 +0000 2021-03-24T20:45:00.533 INFO:teuthology.orchestra.run.smithi063.stdout:Change: 2021-03-24 20:41:46.819906848 +0000 2021-03-24T20:45:00.534 INFO:teuthology.orchestra.run.smithi063.stdout: Birth: - 2021-03-24T20:45:00.541 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records in 2021-03-24T20:45:00.541 INFO:teuthology.orchestra.run.smithi063.stderr:1+0 records out 2021-03-24T20:45:00.542 INFO:teuthology.orchestra.run.smithi063.stderr:512 bytes copied, 0.000400743 s, 1.3 MB/s 2021-03-24T20:45:00.546 INFO:tasks.ceph:osd dev map: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-24T20:45:00.546 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi063.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-03-24T20:45:00.547 INFO:tasks.ceph:Generating config... 2021-03-24T20:45:00.579 INFO:tasks.ceph:[client] debug rgw = 20 2021-03-24T20:45:00.580 INFO:tasks.ceph:[client] rgw crypt require ssl = False 2021-03-24T20:45:00.580 INFO:tasks.ceph:[client] rgw crypt s3 kms backend = testing 2021-03-24T20:45:00.580 INFO:tasks.ceph:[client] rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-24T20:45:00.581 INFO:tasks.ceph:[client] rgw curl low speed time = 300 2021-03-24T20:45:00.581 INFO:tasks.ceph:[client] rgw data log num shards = 4 2021-03-24T20:45:00.581 INFO:tasks.ceph:[client] rgw md log max shards = 4 2021-03-24T20:45:00.582 INFO:tasks.ceph:[client] rgw sync log trim interval = 0 2021-03-24T20:45:00.582 INFO:tasks.ceph:[client] rgw sync obj etag verify = True 2021-03-24T20:45:00.582 INFO:tasks.ceph:[client] setgroup = ceph 2021-03-24T20:45:00.583 INFO:tasks.ceph:[client] setuser = ceph 2021-03-24T20:45:00.583 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-03-24T20:45:00.583 INFO:tasks.ceph:[mgr] debug ms = 1 2021-03-24T20:45:00.583 INFO:tasks.ceph:[mon] debug mon = 20 2021-03-24T20:45:00.584 INFO:tasks.ceph:[mon] debug ms = 1 2021-03-24T20:45:00.584 INFO:tasks.ceph:[mon] debug paxos = 20 2021-03-24T20:45:00.584 INFO:tasks.ceph:[osd] debug ms = 1 2021-03-24T20:45:00.585 INFO:tasks.ceph:[osd] debug osd = 20 2021-03-24T20:45:00.585 INFO:tasks.ceph:Setting up c2.mon.a... 2021-03-24T20:45:00.586 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/c2.keyring 2021-03-24T20:45:00.620 INFO:teuthology.orchestra.run.smithi063.stdout:creating /etc/ceph/c2.keyring 2021-03-24T20:45:00.623 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/c2.keyring 2021-03-24T20:45:00.700 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-03-24T20:45:00.754 DEBUG:tasks.ceph:Ceph mon addresses: [('c2.mon.a', '172.21.15.63')] 2021-03-24T20:45:00.755 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.63'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'debug ms': 1, 'debug osd': 20}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rgw': 20, 'rgw crypt require ssl': False, 'rgw crypt s3 kms backend': 'testing', 'rgw crypt s3 kms encryption keys': 'testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo=', 'rgw curl low speed time': 300, 'rgw data log num shards': 4, 'rgw md log max shards': 4, 'rgw sync log trim interval': 0, 'rgw sync obj etag verify': True, 'setgroup': 'ceph', 'setuser': 'ceph'}, 'mon.a': {}} 2021-03-24T20:45:00.756 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:45:00.757 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-03-24T20:45:00.803 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.63 --print /home/ubuntu/cephtest/c2.monmap 2021-03-24T20:45:00.878 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:45:00.879 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:45:00.881 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: monmap file /home/ubuntu/cephtest/c2.monmap 2021-03-24T20:45:00.881 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: generated fsid 19ff7635-a2a7-4347-a6a0-ab464f77d5fb 2021-03-24T20:45:00.881 INFO:teuthology.orchestra.run.smithi063.stdout:epoch 0 2021-03-24T20:45:00.882 INFO:teuthology.orchestra.run.smithi063.stdout:fsid 19ff7635-a2a7-4347-a6a0-ab464f77d5fb 2021-03-24T20:45:00.882 INFO:teuthology.orchestra.run.smithi063.stdout:last_changed 2021-03-24T20:45:00.881458+0000 2021-03-24T20:45:00.883 INFO:teuthology.orchestra.run.smithi063.stdout:created 2021-03-24T20:45:00.881458+0000 2021-03-24T20:45:00.883 INFO:teuthology.orchestra.run.smithi063.stdout:min_mon_release 0 (unknown) 2021-03-24T20:45:00.884 INFO:teuthology.orchestra.run.smithi063.stdout:0: [v2:172.21.15.63:3300/0,v1:172.21.15.63:6789/0] mon.a 2021-03-24T20:45:00.885 INFO:teuthology.orchestra.run.smithi063.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/c2.monmap (1 monitors) 2021-03-24T20:45:00.885 DEBUG:teuthology.orchestra.run.smithi063:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-03-24T20:45:00.931 INFO:tasks.ceph:Writing /etc/ceph/c2.conf for FSID 19ff7635-a2a7-4347-a6a0-ab464f77d5fb... 2021-03-24T20:45:00.932 DEBUG:teuthology.orchestra.run.smithi052:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-03-24T20:45:00.935 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/c2.conf && sudo chmod 0644 /etc/ceph/c2.conf > /dev/null 2021-03-24T20:45:00.975 INFO:teuthology.orchestra.run.smithi052.stdout:[global] 2021-03-24T20:45:00.975 INFO:teuthology.orchestra.run.smithi052.stdout: chdir = "" 2021-03-24T20:45:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-24T20:45:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: auth supported = cephx 2021-03-24T20:45:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: filestore xattr use omap = true 2021-03-24T20:45:00.976 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: mon clock drift allowed = 1.000 2021-03-24T20:45:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: osd crush chooseleaf type = 0 2021-03-24T20:45:00.977 INFO:teuthology.orchestra.run.smithi052.stdout: auth debug = true 2021-03-24T20:45:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on old message = true 2021-03-24T20:45:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: ms die on bug = true 2021-03-24T20:45:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.978 INFO:teuthology.orchestra.run.smithi052.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-24T20:45:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: mon pg warn max object skew = 0 2021-03-24T20:45:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: # disable pg_autoscaler by default for new pools 2021-03-24T20:45:00.979 INFO:teuthology.orchestra.run.smithi052.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-24T20:45:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default size = 2 2021-03-24T20:45:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow primary affinity = true 2021-03-24T20:45:00.980 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd allow pg remap = true 2021-03-24T20:45:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on legacy crush tunables = false 2021-03-24T20:45:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on crush straw calc version zero = false 2021-03-24T20:45:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on no sortbitwise = false 2021-03-24T20:45:00.981 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on osd down out interval zero = false 2021-03-24T20:45:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: mon warn on too few osds = false 2021-03-24T20:45:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-24T20:45:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-24T20:45:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: mon_allow_pool_size_one = true 2021-03-24T20:45:00.982 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-24T20:45:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: osd default data pool replay window = 5 2021-03-24T20:45:00.983 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.984 INFO:teuthology.orchestra.run.smithi052.stdout: mon allow pool delete = true 2021-03-24T20:45:00.984 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.984 INFO:teuthology.orchestra.run.smithi052.stdout: mon cluster log file level = debug 2021-03-24T20:45:00.984 INFO:teuthology.orchestra.run.smithi052.stdout: debug asserts on shutdown = true 2021-03-24T20:45:00.984 INFO:teuthology.orchestra.run.smithi052.stdout: mon health detail to clog = false 2021-03-24T20:45:00.985 INFO:teuthology.orchestra.run.smithi052.stdout: mon host = 172.21.15.63 2021-03-24T20:45:00.985 INFO:teuthology.orchestra.run.smithi052.stdout: fsid = 19ff7635-a2a7-4347-a6a0-ab464f77d5fb 2021-03-24T20:45:00.985 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.985 INFO:teuthology.orchestra.run.smithi052.stdout:[osd] 2021-03-24T20:45:00.985 INFO:teuthology.orchestra.run.smithi052.stdout: osd journal size = 100 2021-03-24T20:45:00.986 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.986 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub load threshold = 5.0 2021-03-24T20:45:00.986 INFO:teuthology.orchestra.run.smithi052.stdout: osd scrub max interval = 600 2021-03-24T20:45:00.986 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.987 INFO:teuthology.orchestra.run.smithi052.stdout: osd recover clone overlap = true 2021-03-24T20:45:00.987 INFO:teuthology.orchestra.run.smithi052.stdout: osd recovery max chunk = 1048576 2021-03-24T20:45:00.987 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.987 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug shutdown = true 2021-03-24T20:45:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug op order = true 2021-03-24T20:45:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug verify stray on activate = true 2021-03-24T20:45:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: osd open classes on start = true 2021-03-24T20:45:00.988 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug pg log writeout = true 2021-03-24T20:45:00.989 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.989 INFO:teuthology.orchestra.run.smithi052.stdout: osd deep scrub update digest min age = 30 2021-03-24T20:45:00.989 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.989 INFO:teuthology.orchestra.run.smithi052.stdout: osd map max advance = 10 2021-03-24T20:45:00.989 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.990 INFO:teuthology.orchestra.run.smithi052.stdout: journal zero on create = true 2021-03-24T20:45:00.990 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.990 INFO:teuthology.orchestra.run.smithi052.stdout: filestore ondisk finisher threads = 3 2021-03-24T20:45:00.991 INFO:teuthology.orchestra.run.smithi052.stdout: filestore apply finisher threads = 3 2021-03-24T20:45:00.991 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.991 INFO:teuthology.orchestra.run.smithi052.stdout: bdev debug aio = true 2021-03-24T20:45:00.991 INFO:teuthology.orchestra.run.smithi052.stdout: osd debug misdirected ops = true 2021-03-24T20:45:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2021-03-24T20:45:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: debug osd = 20 2021-03-24T20:45:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.992 INFO:teuthology.orchestra.run.smithi052.stdout:[mgr] 2021-03-24T20:45:00.992 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2021-03-24T20:45:00.993 INFO:teuthology.orchestra.run.smithi052.stdout: debug mgr = 20 2021-03-24T20:45:00.993 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2021-03-24T20:45:00.993 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2021-03-24T20:45:00.993 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2021-03-24T20:45:00.994 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2021-03-24T20:45:00.994 INFO:teuthology.orchestra.run.smithi052.stdout: mgr/telemetry/nag = false 2021-03-24T20:45:00.994 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.994 INFO:teuthology.orchestra.run.smithi052.stdout:[mon] 2021-03-24T20:45:00.994 INFO:teuthology.orchestra.run.smithi052.stdout: debug ms = 1 2021-03-24T20:45:00.995 INFO:teuthology.orchestra.run.smithi052.stdout: debug mon = 20 2021-03-24T20:45:00.995 INFO:teuthology.orchestra.run.smithi052.stdout: debug paxos = 20 2021-03-24T20:45:00.995 INFO:teuthology.orchestra.run.smithi052.stdout: debug auth = 20 2021-03-24T20:45:00.995 INFO:teuthology.orchestra.run.smithi052.stdout: mon data avail warn = 5 2021-03-24T20:45:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: mon mgr mkfs grace = 240 2021-03-24T20:45:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min pgs per osd = 4 2021-03-24T20:45:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd reporter subtree level = osd 2021-03-24T20:45:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: mon osd prime pg temp = true 2021-03-24T20:45:00.996 INFO:teuthology.orchestra.run.smithi052.stdout: mon reweight min bytes per osd = 10 2021-03-24T20:45:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T20:45:00.997 INFO:teuthology.orchestra.run.smithi052.stdout:[client] 2021-03-24T20:45:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: rgw cache enabled = true 2021-03-24T20:45:00.997 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable ops log = true 2021-03-24T20:45:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: rgw enable usage log = true 2021-03-24T20:45:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-24T20:45:00.998 INFO:teuthology.orchestra.run.smithi052.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-24T20:45:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: debug rgw = 20 2021-03-24T20:45:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt require ssl = False 2021-03-24T20:45:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt s3 kms backend = testing 2021-03-24T20:45:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-24T20:45:00.999 INFO:teuthology.orchestra.run.smithi052.stdout: rgw curl low speed time = 300 2021-03-24T20:45:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: rgw data log num shards = 4 2021-03-24T20:45:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: rgw md log max shards = 4 2021-03-24T20:45:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: rgw sync log trim interval = 0 2021-03-24T20:45:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: rgw sync obj etag verify = True 2021-03-24T20:45:01.000 INFO:teuthology.orchestra.run.smithi052.stdout: setgroup = ceph 2021-03-24T20:45:01.001 INFO:teuthology.orchestra.run.smithi052.stdout: setuser = ceph 2021-03-24T20:45:01.001 INFO:teuthology.orchestra.run.smithi052.stdout:[mon.a] 2021-03-24T20:45:01.007 INFO:teuthology.orchestra.run.smithi063.stdout:[global] 2021-03-24T20:45:01.007 INFO:teuthology.orchestra.run.smithi063.stdout: chdir = "" 2021-03-24T20:45:01.007 INFO:teuthology.orchestra.run.smithi063.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-03-24T20:45:01.008 INFO:teuthology.orchestra.run.smithi063.stdout: auth supported = cephx 2021-03-24T20:45:01.008 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.008 INFO:teuthology.orchestra.run.smithi063.stdout: filestore xattr use omap = true 2021-03-24T20:45:01.009 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.009 INFO:teuthology.orchestra.run.smithi063.stdout: mon clock drift allowed = 1.000 2021-03-24T20:45:01.009 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.009 INFO:teuthology.orchestra.run.smithi063.stdout: osd crush chooseleaf type = 0 2021-03-24T20:45:01.009 INFO:teuthology.orchestra.run.smithi063.stdout: auth debug = true 2021-03-24T20:45:01.010 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.010 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on old message = true 2021-03-24T20:45:01.010 INFO:teuthology.orchestra.run.smithi063.stdout: ms die on bug = true 2021-03-24T20:45:01.010 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.011 INFO:teuthology.orchestra.run.smithi063.stdout: mon max pg per osd = 10000 # >= luminous 2021-03-24T20:45:01.011 INFO:teuthology.orchestra.run.smithi063.stdout: mon pg warn max object skew = 0 2021-03-24T20:45:01.011 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.011 INFO:teuthology.orchestra.run.smithi063.stdout: # disable pg_autoscaler by default for new pools 2021-03-24T20:45:01.012 INFO:teuthology.orchestra.run.smithi063.stdout: osd_pool_default_pg_autoscale_mode = off 2021-03-24T20:45:01.012 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.012 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default size = 2 2021-03-24T20:45:01.013 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.013 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow primary affinity = true 2021-03-24T20:45:01.013 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd allow pg remap = true 2021-03-24T20:45:01.013 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on legacy crush tunables = false 2021-03-24T20:45:01.014 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on crush straw calc version zero = false 2021-03-24T20:45:01.014 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on no sortbitwise = false 2021-03-24T20:45:01.014 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on osd down out interval zero = false 2021-03-24T20:45:01.014 INFO:teuthology.orchestra.run.smithi063.stdout: mon warn on too few osds = false 2021-03-24T20:45:01.014 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-03-24T20:45:01.015 INFO:teuthology.orchestra.run.smithi063.stdout: mon_warn_on_pool_no_redundancy = false 2021-03-24T20:45:01.015 INFO:teuthology.orchestra.run.smithi063.stdout: mon_allow_pool_size_one = true 2021-03-24T20:45:01.015 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.015 INFO:teuthology.orchestra.run.smithi063.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-03-24T20:45:01.016 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.016 INFO:teuthology.orchestra.run.smithi063.stdout: osd default data pool replay window = 5 2021-03-24T20:45:01.016 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.016 INFO:teuthology.orchestra.run.smithi063.stdout: mon allow pool delete = true 2021-03-24T20:45:01.017 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.017 INFO:teuthology.orchestra.run.smithi063.stdout: mon cluster log file level = debug 2021-03-24T20:45:01.017 INFO:teuthology.orchestra.run.smithi063.stdout: debug asserts on shutdown = true 2021-03-24T20:45:01.017 INFO:teuthology.orchestra.run.smithi063.stdout: mon health detail to clog = false 2021-03-24T20:45:01.018 INFO:teuthology.orchestra.run.smithi063.stdout: mon host = 172.21.15.63 2021-03-24T20:45:01.018 INFO:teuthology.orchestra.run.smithi063.stdout: fsid = 19ff7635-a2a7-4347-a6a0-ab464f77d5fb 2021-03-24T20:45:01.018 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.018 INFO:teuthology.orchestra.run.smithi063.stdout:[osd] 2021-03-24T20:45:01.019 INFO:teuthology.orchestra.run.smithi063.stdout: osd journal size = 100 2021-03-24T20:45:01.019 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.019 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub load threshold = 5.0 2021-03-24T20:45:01.020 INFO:teuthology.orchestra.run.smithi063.stdout: osd scrub max interval = 600 2021-03-24T20:45:01.020 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.020 INFO:teuthology.orchestra.run.smithi063.stdout: osd recover clone overlap = true 2021-03-24T20:45:01.020 INFO:teuthology.orchestra.run.smithi063.stdout: osd recovery max chunk = 1048576 2021-03-24T20:45:01.020 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.021 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug shutdown = true 2021-03-24T20:45:01.021 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug op order = true 2021-03-24T20:45:01.021 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug verify stray on activate = true 2021-03-24T20:45:01.021 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.022 INFO:teuthology.orchestra.run.smithi063.stdout: osd open classes on start = true 2021-03-24T20:45:01.022 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug pg log writeout = true 2021-03-24T20:45:01.022 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.022 INFO:teuthology.orchestra.run.smithi063.stdout: osd deep scrub update digest min age = 30 2021-03-24T20:45:01.022 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.023 INFO:teuthology.orchestra.run.smithi063.stdout: osd map max advance = 10 2021-03-24T20:45:01.023 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.023 INFO:teuthology.orchestra.run.smithi063.stdout: journal zero on create = true 2021-03-24T20:45:01.023 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.024 INFO:teuthology.orchestra.run.smithi063.stdout: filestore ondisk finisher threads = 3 2021-03-24T20:45:01.024 INFO:teuthology.orchestra.run.smithi063.stdout: filestore apply finisher threads = 3 2021-03-24T20:45:01.024 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.024 INFO:teuthology.orchestra.run.smithi063.stdout: bdev debug aio = true 2021-03-24T20:45:01.024 INFO:teuthology.orchestra.run.smithi063.stdout: osd debug misdirected ops = true 2021-03-24T20:45:01.025 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2021-03-24T20:45:01.025 INFO:teuthology.orchestra.run.smithi063.stdout: debug osd = 20 2021-03-24T20:45:01.025 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.025 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr] 2021-03-24T20:45:01.026 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2021-03-24T20:45:01.026 INFO:teuthology.orchestra.run.smithi063.stdout: debug mgr = 20 2021-03-24T20:45:01.026 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2021-03-24T20:45:01.026 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2021-03-24T20:45:01.026 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2021-03-24T20:45:01.027 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2021-03-24T20:45:01.027 INFO:teuthology.orchestra.run.smithi063.stdout: mgr/telemetry/nag = false 2021-03-24T20:45:01.027 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.027 INFO:teuthology.orchestra.run.smithi063.stdout:[mon] 2021-03-24T20:45:01.028 INFO:teuthology.orchestra.run.smithi063.stdout: debug ms = 1 2021-03-24T20:45:01.028 INFO:teuthology.orchestra.run.smithi063.stdout: debug mon = 20 2021-03-24T20:45:01.028 INFO:teuthology.orchestra.run.smithi063.stdout: debug paxos = 20 2021-03-24T20:45:01.028 INFO:teuthology.orchestra.run.smithi063.stdout: debug auth = 20 2021-03-24T20:45:01.028 INFO:teuthology.orchestra.run.smithi063.stdout: mon data avail warn = 5 2021-03-24T20:45:01.030 INFO:teuthology.orchestra.run.smithi063.stdout: mon mgr mkfs grace = 240 2021-03-24T20:45:01.030 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min pgs per osd = 4 2021-03-24T20:45:01.030 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd reporter subtree level = osd 2021-03-24T20:45:01.030 INFO:teuthology.orchestra.run.smithi063.stdout: mon osd prime pg temp = true 2021-03-24T20:45:01.031 INFO:teuthology.orchestra.run.smithi063.stdout: mon reweight min bytes per osd = 10 2021-03-24T20:45:01.031 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:01.031 INFO:teuthology.orchestra.run.smithi063.stdout:[client] 2021-03-24T20:45:01.031 INFO:teuthology.orchestra.run.smithi063.stdout: rgw cache enabled = true 2021-03-24T20:45:01.032 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable ops log = true 2021-03-24T20:45:01.032 INFO:teuthology.orchestra.run.smithi063.stdout: rgw enable usage log = true 2021-03-24T20:45:01.032 INFO:teuthology.orchestra.run.smithi063.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-03-24T20:45:01.032 INFO:teuthology.orchestra.run.smithi063.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-03-24T20:45:01.032 INFO:teuthology.orchestra.run.smithi063.stdout: debug rgw = 20 2021-03-24T20:45:01.033 INFO:teuthology.orchestra.run.smithi063.stdout: rgw crypt require ssl = False 2021-03-24T20:45:01.033 INFO:teuthology.orchestra.run.smithi063.stdout: rgw crypt s3 kms backend = testing 2021-03-24T20:45:01.033 INFO:teuthology.orchestra.run.smithi063.stdout: rgw crypt s3 kms encryption keys = testkey-1=YmluCmJvb3N0CmJvb3N0LWJ1aWxkCmNlcGguY29uZgo= 2021-03-24T20:45:01.033 INFO:teuthology.orchestra.run.smithi063.stdout: rgw curl low speed time = 300 2021-03-24T20:45:01.034 INFO:teuthology.orchestra.run.smithi063.stdout: rgw data log num shards = 4 2021-03-24T20:45:01.034 INFO:teuthology.orchestra.run.smithi063.stdout: rgw md log max shards = 4 2021-03-24T20:45:01.034 INFO:teuthology.orchestra.run.smithi063.stdout: rgw sync log trim interval = 0 2021-03-24T20:45:01.035 INFO:teuthology.orchestra.run.smithi063.stdout: rgw sync obj etag verify = True 2021-03-24T20:45:01.035 INFO:teuthology.orchestra.run.smithi063.stdout: setgroup = ceph 2021-03-24T20:45:01.035 INFO:teuthology.orchestra.run.smithi063.stdout: setuser = ceph 2021-03-24T20:45:01.035 INFO:teuthology.orchestra.run.smithi063.stdout:[mon.a] 2021-03-24T20:45:01.037 INFO:tasks.ceph:Creating admin key on c2.mon.a... 2021-03-24T20:45:01.037 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/c2.keyring 2021-03-24T20:45:01.126 INFO:tasks.ceph:Copying monmap to all nodes... 2021-03-24T20:45:01.127 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:45:01.127 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/c2.keyring of=/dev/stdout 2021-03-24T20:45:01.134 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:45:01.135 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/home/ubuntu/cephtest/c2.monmap of=/dev/stdout 2021-03-24T20:45:01.184 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi052.front.sepia.ceph.com 2021-03-24T20:45:01.184 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:45:01.185 DEBUG:teuthology.orchestra.run.smithi052:> sudo dd of=/etc/ceph/c2.keyring 2021-03-24T20:45:01.186 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-03-24T20:45:01.213 DEBUG:teuthology.orchestra.run.smithi052:> set -ex 2021-03-24T20:45:01.214 DEBUG:teuthology.orchestra.run.smithi052:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-03-24T20:45:01.265 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi063.front.sepia.ceph.com 2021-03-24T20:45:01.266 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:45:01.266 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/ceph/c2.keyring 2021-03-24T20:45:01.267 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0644 /etc/ceph/c2.keyring 2021-03-24T20:45:01.293 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:45:01.294 DEBUG:teuthology.orchestra.run.smithi063:> dd of=/home/ubuntu/cephtest/c2.monmap 2021-03-24T20:45:01.347 INFO:tasks.ceph:Setting up mon nodes... 2021-03-24T20:45:01.349 INFO:tasks.ceph:Setting up mgr nodes... 2021-03-24T20:45:01.349 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mgr/c2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/c2-x/keyring 2021-03-24T20:45:01.433 INFO:teuthology.orchestra.run.smithi063.stdout:creating /var/lib/ceph/mgr/c2-x/keyring 2021-03-24T20:45:01.435 INFO:tasks.ceph:Setting up mds nodes... 2021-03-24T20:45:01.436 INFO:tasks.ceph_client:Setting up client nodes... 2021-03-24T20:45:01.438 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/c2.client.0.keyring && sudo chmod 0644 /etc/ceph/c2.client.0.keyring 2021-03-24T20:45:01.510 INFO:teuthology.orchestra.run.smithi063.stdout:creating /etc/ceph/c2.client.0.keyring 2021-03-24T20:45:01.522 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/c2.client.1.keyring && sudo chmod 0644 /etc/ceph/c2.client.1.keyring 2021-03-24T20:45:01.597 INFO:teuthology.orchestra.run.smithi063.stdout:creating /etc/ceph/c2.client.1.keyring 2021-03-24T20:45:01.610 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-03-24T20:45:01.611 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi052.front.sepia.ceph.com'): {'c1.osd.0': '/dev/vg_nvme/lv_1', 'c1.osd.1': '/dev/vg_nvme/lv_2', 'c1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi063.front.sepia.ceph.com'): {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'}} 2021-03-24T20:45:01.611 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/c2-0 2021-03-24T20:45:01.667 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-24T20:45:01.668 INFO:tasks.ceph:role: c2.osd.0 2021-03-24T20:45:01.668 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi063.front.sepia.ceph.com 2021-03-24T20:45:01.669 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-03-24T20:45:02.172 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-03-24T20:45:02.173 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-24T20:45:02.173 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-24T20:45:02.173 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-24T20:45:02.174 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2021-03-24T20:45:02.174 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-24T20:45:02.174 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-24T20:45:02.175 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-24T20:45:02.175 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-24T20:45:02.177 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2021-03-24T20:45:02.178 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/c2-0 2021-03-24T20:45:02.254 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-0 2021-03-24T20:45:02.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T20:45:02.268 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: /sbin/restorecon: command not found 2021-03-24T20:45:02.268 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/c2-1 2021-03-24T20:45:02.323 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-24T20:45:02.323 INFO:tasks.ceph:role: c2.osd.1 2021-03-24T20:45:02.324 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi063.front.sepia.ceph.com 2021-03-24T20:45:02.325 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-03-24T20:45:02.843 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-03-24T20:45:02.843 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-24T20:45:02.844 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-24T20:45:02.844 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-24T20:45:02.844 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2021-03-24T20:45:02.844 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-24T20:45:02.845 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-24T20:45:02.845 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-24T20:45:02.845 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-24T20:45:02.846 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2021-03-24T20:45:02.847 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/c2-1 2021-03-24T20:45:02.919 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-1 2021-03-24T20:45:02.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T20:45:02.933 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: /sbin/restorecon: command not found 2021-03-24T20:45:02.934 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/osd/c2-2 2021-03-24T20:45:02.992 INFO:tasks.ceph:roles_to_devs: {'c2.osd.0': '/dev/vg_nvme/lv_1', 'c2.osd.1': '/dev/vg_nvme/lv_2', 'c2.osd.2': '/dev/vg_nvme/lv_3'} 2021-03-24T20:45:02.992 INFO:tasks.ceph:role: c2.osd.2 2021-03-24T20:45:02.992 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi063.front.sepia.ceph.com 2021-03-24T20:45:02.993 DEBUG:teuthology.orchestra.run.smithi063:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-03-24T20:45:03.486 INFO:teuthology.orchestra.run.smithi063.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-03-24T20:45:03.487 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 attr=2, projid32bit=1 2021-03-24T20:45:03.487 INFO:teuthology.orchestra.run.smithi063.stdout: = crc=1 finobt=1, sparse=0, rmapbt=0, reflink=0 2021-03-24T20:45:03.487 INFO:teuthology.orchestra.run.smithi063.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-03-24T20:45:03.488 INFO:teuthology.orchestra.run.smithi063.stdout: = sunit=0 swidth=0 blks 2021-03-24T20:45:03.488 INFO:teuthology.orchestra.run.smithi063.stdout:naming =version 2 bsize=4096 ascii-ci=0 ftype=1 2021-03-24T20:45:03.488 INFO:teuthology.orchestra.run.smithi063.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-03-24T20:45:03.489 INFO:teuthology.orchestra.run.smithi063.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-03-24T20:45:03.489 INFO:teuthology.orchestra.run.smithi063.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-03-24T20:45:03.490 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi063.front.sepia.ceph.com -o noatime 2021-03-24T20:45:03.491 DEBUG:teuthology.orchestra.run.smithi063:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/c2-2 2021-03-24T20:45:03.545 DEBUG:teuthology.orchestra.run.smithi063:> sudo /sbin/restorecon /var/lib/ceph/osd/c2-2 2021-03-24T20:45:03.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T20:45:03.558 INFO:teuthology.orchestra.run.smithi063.stderr:sudo: /sbin/restorecon: command not found 2021-03-24T20:45:03.559 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/c2.monmap 2021-03-24T20:45:03.662 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:03.663+0000 7f69a26fed80 -1 auth: error reading file: /var/lib/ceph/osd/c2-0/keyring: can't open /var/lib/ceph/osd/c2-0/keyring: (2) No such file or directory 2021-03-24T20:45:03.663 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:03.663+0000 7f69a26fed80 -1 created new key in keyring /var/lib/ceph/osd/c2-0/keyring 2021-03-24T20:45:03.663 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:03.663+0000 7f69a26fed80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-03-24T20:45:03.663 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:03.663+0000 7f69a26fed80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-03-24T20:45:03.663 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:03.663+0000 7f69a26fed80 -1 bluestore(/var/lib/ceph/osd/c2-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-0/block: (2) No such file or directory 2021-03-24T20:45:03.665 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:03.667+0000 7f69a26fed80 -1 bluestore(/var/lib/ceph/osd/c2-0) _read_fsid unparsable uuid 2021-03-24T20:45:03.691 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:03.691+0000 7f69a26fed80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-24T20:45:05.568 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-0 2021-03-24T20:45:05.585 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/c2.monmap 2021-03-24T20:45:05.667 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:05.667+0000 7f1f5f5e9d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-1/keyring: can't open /var/lib/ceph/osd/c2-1/keyring: (2) No such file or directory 2021-03-24T20:45:05.668 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:05.667+0000 7f1f5f5e9d80 -1 created new key in keyring /var/lib/ceph/osd/c2-1/keyring 2021-03-24T20:45:05.668 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:05.667+0000 7f1f5f5e9d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-03-24T20:45:05.668 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:05.667+0000 7f1f5f5e9d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-03-24T20:45:05.669 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:05.667+0000 7f1f5f5e9d80 -1 bluestore(/var/lib/ceph/osd/c2-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-1/block: (2) No such file or directory 2021-03-24T20:45:05.669 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:05.667+0000 7f1f5f5e9d80 -1 bluestore(/var/lib/ceph/osd/c2-1) _read_fsid unparsable uuid 2021-03-24T20:45:05.683 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:05.683+0000 7f1f5f5e9d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-24T20:45:07.575 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-1 2021-03-24T20:45:07.632 DEBUG:teuthology.orchestra.run.smithi063:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster c2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/c2.monmap 2021-03-24T20:45:07.715 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:07.715+0000 7fb5bb487d80 -1 auth: error reading file: /var/lib/ceph/osd/c2-2/keyring: can't open /var/lib/ceph/osd/c2-2/keyring: (2) No such file or directory 2021-03-24T20:45:07.715 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:07.715+0000 7fb5bb487d80 -1 created new key in keyring /var/lib/ceph/osd/c2-2/keyring 2021-03-24T20:45:07.715 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:07.715+0000 7fb5bb487d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-03-24T20:45:07.716 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:07.715+0000 7fb5bb487d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-03-24T20:45:07.716 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:07.715+0000 7fb5bb487d80 -1 bluestore(/var/lib/ceph/osd/c2-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/c2-2/block: (2) No such file or directory 2021-03-24T20:45:07.716 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:07.715+0000 7fb5bb487d80 -1 bluestore(/var/lib/ceph/osd/c2-2) _read_fsid unparsable uuid 2021-03-24T20:45:07.729 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:07.731+0000 7fb5bb487d80 -1 freelist read_size_meta_from_db missing size meta in DB 2021-03-24T20:45:09.639 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/osd/c2-2 2021-03-24T20:45:09.697 INFO:tasks.ceph:Reading keys from all nodes... 2021-03-24T20:45:09.697 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:45:09.697 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/mgr/c2-x/keyring of=/dev/stdout 2021-03-24T20:45:09.755 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:45:09.756 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/c2-0/keyring of=/dev/stdout 2021-03-24T20:45:09.810 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:45:09.811 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/c2-1/keyring of=/dev/stdout 2021-03-24T20:45:09.866 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:45:09.867 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/c2-2/keyring of=/dev/stdout 2021-03-24T20:45:09.922 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:45:09.923 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/c2.client.0.keyring of=/dev/stdout 2021-03-24T20:45:09.970 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:45:09.971 DEBUG:teuthology.orchestra.run.smithi063:> dd if=/etc/ceph/c2.client.1.keyring of=/dev/stdout 2021-03-24T20:45:10.021 INFO:tasks.ceph:Adding keys to all mons... 2021-03-24T20:45:10.022 DEBUG:teuthology.orchestra.run.smithi063:> sudo tee -a /etc/ceph/c2.keyring 2021-03-24T20:45:10.077 INFO:teuthology.orchestra.run.smithi063.stdout:[mgr.x] 2021-03-24T20:45:10.078 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDNpFtgXGv4GRAA0ZGAV0z5AV0fgcb48XRtTQ== 2021-03-24T20:45:10.078 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.0] 2021-03-24T20:45:10.078 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDPpFtgAGOYJxAA0fuocec4tcU8Mb+5/3U8AA== 2021-03-24T20:45:10.079 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.1] 2021-03-24T20:45:10.079 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDRpFtg67zlJxAAEGfT4oE/Px1hDqMml3CRsA== 2021-03-24T20:45:10.079 INFO:teuthology.orchestra.run.smithi063.stdout:[osd.2] 2021-03-24T20:45:10.079 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDTpFtgIhu7KhAA12M7f4hy4BDtx8sYwUlZ0A== 2021-03-24T20:45:10.080 INFO:teuthology.orchestra.run.smithi063.stdout:[client.0] 2021-03-24T20:45:10.080 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDNpFtgQCSOHhAAk0kF4rmiCaJ7JZMfZx86xA== 2021-03-24T20:45:10.080 INFO:teuthology.orchestra.run.smithi063.stdout:[client.1] 2021-03-24T20:45:10.080 INFO:teuthology.orchestra.run.smithi063.stdout: key = AQDNpFtg24HHIxAAzxrTYKvrKAFm4Wv6PeqZbg== 2021-03-24T20:45:10.081 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-03-24T20:45:10.167 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-24T20:45:10.255 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-24T20:45:10.340 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-03-24T20:45:10.427 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-24T20:45:10.513 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/c2.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2021-03-24T20:45:10.559 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-03-24T20:45:10.560 DEBUG:teuthology.orchestra.run.smithi063:> sudo mkdir -p /var/lib/ceph/mon/c2-a 2021-03-24T20:45:10.615 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster c2 --mkfs -i a --monmap /home/ubuntu/cephtest/c2.monmap --keyring /etc/ceph/c2.keyring 2021-03-24T20:45:10.730 DEBUG:teuthology.orchestra.run.smithi063:> sudo chown -R ceph:ceph /var/lib/ceph/mon/c2-a 2021-03-24T20:45:10.746 DEBUG:teuthology.orchestra.run.smithi063:> rm -- /home/ubuntu/cephtest/c2.monmap 2021-03-24T20:45:10.794 INFO:tasks.ceph:Starting mon daemons in cluster c2... 2021-03-24T20:45:10.794 INFO:tasks.ceph.c2.mon.a:Restarting daemon 2021-03-24T20:45:10.795 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster c2 -i a 2021-03-24T20:45:10.838 INFO:tasks.ceph.c2.mon.a:Started 2021-03-24T20:45:10.838 INFO:tasks.ceph:Starting mgr daemons in cluster c2... 2021-03-24T20:45:10.839 INFO:tasks.ceph.c2.mgr.x:Restarting daemon 2021-03-24T20:45:10.839 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster c2 -i x 2021-03-24T20:45:10.841 INFO:tasks.ceph.c2.mgr.x:Started 2021-03-24T20:45:10.842 INFO:tasks.ceph:Setting crush tunables to default 2021-03-24T20:45:10.842 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster c2 osd crush tunables default 2021-03-24T20:45:11.162 INFO:teuthology.orchestra.run.smithi063.stderr:adjusted tunables profile to default 2021-03-24T20:45:11.173 INFO:tasks.ceph:Starting osd daemons in cluster c2... 2021-03-24T20:45:11.174 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:45:11.174 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/c2-0/fsid of=/dev/stdout 2021-03-24T20:45:11.185 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:45:11.185 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/c2-1/fsid of=/dev/stdout 2021-03-24T20:45:11.241 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:45:11.241 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd if=/var/lib/ceph/osd/c2-2/fsid of=/dev/stdout 2021-03-24T20:45:11.298 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster c2 osd new 4716e8a9-b4a7-4c50-a4e7-8c860600febd 0 2021-03-24T20:45:11.548 INFO:teuthology.orchestra.run.smithi063.stdout:0 2021-03-24T20:45:11.561 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster c2 osd new e72ecbde-5057-4f63-b5ce-ce1fe8f0cfa4 1 2021-03-24T20:45:11.760 INFO:teuthology.orchestra.run.smithi063.stdout:1 2021-03-24T20:45:11.771 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster c2 osd new d87e26ae-d46d-4786-b961-8369e589f488 2 2021-03-24T20:45:11.976 INFO:teuthology.orchestra.run.smithi063.stdout:2 2021-03-24T20:45:11.989 INFO:tasks.ceph.c2.osd.0:Restarting daemon 2021-03-24T20:45:11.990 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 0 2021-03-24T20:45:11.992 INFO:tasks.ceph.c2.osd.0:Started 2021-03-24T20:45:11.993 INFO:tasks.ceph.c2.osd.1:Restarting daemon 2021-03-24T20:45:11.993 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 1 2021-03-24T20:45:11.995 INFO:tasks.ceph.c2.osd.1:Started 2021-03-24T20:45:11.995 INFO:tasks.ceph.c2.osd.2:Restarting daemon 2021-03-24T20:45:11.996 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster c2 -i 2 2021-03-24T20:45:11.998 INFO:tasks.ceph.c2.osd.2:Started 2021-03-24T20:45:11.998 INFO:tasks.ceph:Waiting for OSDs to come up 2021-03-24T20:45:12.301 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-03-24T20:45:12.357 INFO:tasks.ceph.c2.osd.2.smithi063.stderr:2021-03-24T20:45:12.359+0000 7f82d91e3d80 -1 Falling back to public interface 2021-03-24T20:45:12.366 INFO:tasks.ceph.c2.osd.1.smithi063.stderr:2021-03-24T20:45:12.367+0000 7f2eb7d07d80 -1 Falling back to public interface 2021-03-24T20:45:12.464 INFO:teuthology.misc.health.smithi063.stdout: 2021-03-24T20:45:12.465 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":5,"fsid":"19ff7635-a2a7-4347-a6a0-ab464f77d5fb","created":"2021-03-24T20:45:10.980569+0000","modified":"2021-03-24T20:45:11.976522+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[],"osds":[{"osd":0,"uuid":"4716e8a9-b4a7-4c50-a4e7-8c860600febd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"e72ecbde-5057-4f63-b5ce-ce1fe8f0cfa4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"d87e26ae-d46d-4786-b961-8369e589f488","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-24T20:45:12.475 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-03-24T20:45:12.598 INFO:tasks.ceph.c2.osd.0.smithi063.stderr:2021-03-24T20:45:12.599+0000 7f51e80a4d80 -1 Falling back to public interface 2021-03-24T20:45:12.969 INFO:tasks.ceph.c2.osd.2.smithi063.stderr:2021-03-24T20:45:12.971+0000 7f82d91e3d80 -1 osd.2 0 log_to_monitors {default=true} 2021-03-24T20:45:12.988 INFO:tasks.ceph.c2.osd.1.smithi063.stderr:2021-03-24T20:45:12.991+0000 7f2eb7d07d80 -1 osd.1 0 log_to_monitors {default=true} 2021-03-24T20:45:13.206 INFO:tasks.ceph.c2.osd.0.smithi063.stderr:2021-03-24T20:45:13.207+0000 7f51e80a4d80 -1 osd.0 0 log_to_monitors {default=true} 2021-03-24T20:45:14.007 INFO:tasks.ceph.c2.osd.1.smithi063.stderr:2021-03-24T20:45:14.007+0000 7f2ea1753700 -1 osd.1 0 waiting for initial osdmap 2021-03-24T20:45:14.007 INFO:tasks.ceph.c2.osd.2.smithi063.stderr:2021-03-24T20:45:14.007+0000 7f82c2c2f700 -1 osd.2 0 waiting for initial osdmap 2021-03-24T20:45:15.070 INFO:tasks.ceph.c2.osd.0.smithi063.stderr:2021-03-24T20:45:15.071+0000 7f51d1af0700 -1 osd.0 0 waiting for initial osdmap 2021-03-24T20:45:18.778 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster c2 osd dump --format=json 2021-03-24T20:45:19.054 INFO:teuthology.misc.health.smithi063.stdout: 2021-03-24T20:45:19.054 INFO:teuthology.misc.health.smithi063.stdout:{"epoch":11,"fsid":"19ff7635-a2a7-4347-a6a0-ab464f77d5fb","created":"2021-03-24T20:45:10.980569+0000","modified":"2021-03-24T20:45:18.085568+0000","last_up_change":"2021-03-24T20:45:16.069521+0000","last_in_change":"2021-03-24T20:45:16.069521+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-24T20:45:15.056543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"4716e8a9-b4a7-4c50-a4e7-8c860600febd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":12758},{"type":"v1","addr":"172.21.15.63:6817","nonce":12758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":12758},{"type":"v1","addr":"172.21.15.63:6819","nonce":12758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":12758},{"type":"v1","addr":"172.21.15.63:6823","nonce":12758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":12758},{"type":"v1","addr":"172.21.15.63:6821","nonce":12758}]},"public_addr":"172.21.15.63:6817/12758","cluster_addr":"172.21.15.63:6819/12758","heartbeat_back_addr":"172.21.15.63:6823/12758","heartbeat_front_addr":"172.21.15.63:6821/12758","state":["exists","up"]},{"osd":1,"uuid":"e72ecbde-5057-4f63-b5ce-ce1fe8f0cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":12759},{"type":"v1","addr":"172.21.15.63:6809","nonce":12759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":12759},{"type":"v1","addr":"172.21.15.63:6811","nonce":12759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":12759},{"type":"v1","addr":"172.21.15.63:6815","nonce":12759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":12759},{"type":"v1","addr":"172.21.15.63:6813","nonce":12759}]},"public_addr":"172.21.15.63:6809/12759","cluster_addr":"172.21.15.63:6811/12759","heartbeat_back_addr":"172.21.15.63:6815/12759","heartbeat_front_addr":"172.21.15.63:6813/12759","state":["exists","up"]},{"osd":2,"uuid":"d87e26ae-d46d-4786-b961-8369e589f488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":12757},{"type":"v1","addr":"172.21.15.63:6801","nonce":12757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":12757},{"type":"v1","addr":"172.21.15.63:6803","nonce":12757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":12757},{"type":"v1","addr":"172.21.15.63:6807","nonce":12757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":12757},{"type":"v1","addr":"172.21.15.63:6805","nonce":12757}]},"public_addr":"172.21.15.63:6801/12757","cluster_addr":"172.21.15.63:6803/12757","heartbeat_back_addr":"172.21.15.63:6807/12757","heartbeat_front_addr":"172.21.15.63:6805/12757","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:45:14.259698+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:45:13.993033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:45:13.945932+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-24T20:45:19.066 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-03-24T20:45:19.066 INFO:tasks.ceph:Creating RBD pool 2021-03-24T20:45:19.067 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster c2 osd pool create rbd 8 2021-03-24T20:45:20.114 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'rbd' created 2021-03-24T20:45:20.128 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster c2 osd pool application enable rbd rbd --yes-i-really-mean-it 2021-03-24T20:45:21.118 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rbd' on pool 'rbd' 2021-03-24T20:45:21.131 INFO:tasks.ceph:Starting mds daemons in cluster c2... 2021-03-24T20:45:21.132 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-03-24T20:45:21.132 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-03-24T20:45:21.394 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:21.395 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":13,"fsid":"19ff7635-a2a7-4347-a6a0-ab464f77d5fb","created":"2021-03-24T20:45:10.980569+0000","modified":"2021-03-24T20:45:21.104992+0000","last_up_change":"2021-03-24T20:45:16.069521+0000","last_in_change":"2021-03-24T20:45:16.069521+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-24T20:45:15.056543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-24T20:45:19.412470+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"4716e8a9-b4a7-4c50-a4e7-8c860600febd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":12758},{"type":"v1","addr":"172.21.15.63:6817","nonce":12758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":12758},{"type":"v1","addr":"172.21.15.63:6819","nonce":12758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":12758},{"type":"v1","addr":"172.21.15.63:6823","nonce":12758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":12758},{"type":"v1","addr":"172.21.15.63:6821","nonce":12758}]},"public_addr":"172.21.15.63:6817/12758","cluster_addr":"172.21.15.63:6819/12758","heartbeat_back_addr":"172.21.15.63:6823/12758","heartbeat_front_addr":"172.21.15.63:6821/12758","state":["exists","up"]},{"osd":1,"uuid":"e72ecbde-5057-4f63-b5ce-ce1fe8f0cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":12759},{"type":"v1","addr":"172.21.15.63:6809","nonce":12759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":12759},{"type":"v1","addr":"172.21.15.63:6811","nonce":12759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":12759},{"type":"v1","addr":"172.21.15.63:6815","nonce":12759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":12759},{"type":"v1","addr":"172.21.15.63:6813","nonce":12759}]},"public_addr":"172.21.15.63:6809/12759","cluster_addr":"172.21.15.63:6811/12759","heartbeat_back_addr":"172.21.15.63:6815/12759","heartbeat_front_addr":"172.21.15.63:6813/12759","state":["exists","up"]},{"osd":2,"uuid":"d87e26ae-d46d-4786-b961-8369e589f488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":12757},{"type":"v1","addr":"172.21.15.63:6801","nonce":12757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":12757},{"type":"v1","addr":"172.21.15.63:6803","nonce":12757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":12757},{"type":"v1","addr":"172.21.15.63:6807","nonce":12757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":12757},{"type":"v1","addr":"172.21.15.63:6805","nonce":12757}]},"public_addr":"172.21.15.63:6801/12757","cluster_addr":"172.21.15.63:6803/12757","heartbeat_back_addr":"172.21.15.63:6807/12757","heartbeat_front_addr":"172.21.15.63:6805/12757","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:45:14.259698+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:45:13.993033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:45:13.945932+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-24T20:45:21.408 INFO:tasks.ceph.ceph_manager.c2:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2021-03-24T20:45:15.056543+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2021-03-24T20:45:19.412470+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '13', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-03-24T20:45:21.408 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get device_health_metrics pg_num 2021-03-24T20:45:21.690 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 1 2021-03-24T20:45:21.703 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd pool get rbd pg_num 2021-03-24T20:45:21.989 INFO:teuthology.orchestra.run.smithi063.stdout:pg_num: 8 2021-03-24T20:45:22.004 INFO:tasks.ceph:Waiting until c2 daemons up and pgs clean... 2021-03-24T20:45:22.004 INFO:tasks.ceph.ceph_manager.c2:waiting for mgr available 2021-03-24T20:45:22.005 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 mgr dump --format=json 2021-03-24T20:45:22.323 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:22.335 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":4,"active_gid":4102,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6824","nonce":12570},{"type":"v1","addr":"172.21.15.63:6825","nonce":12570}]},"active_addr":"172.21.15.63:6825/12570","active_change":"2021-03-24T20:45:14.960017+0000","active_mgr_features":4540138292840890367,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":false,"error_string":"loading remoto library:No module named 'remoto'","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":2581968206}]},{"addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":3942946809}]},{"addrvec":[{"type":"v2","addr":"172.21.15.63:0","nonce":865796295}]}]}} 2021-03-24T20:45:22.337 INFO:tasks.ceph.ceph_manager.c2:mgr available! 2021-03-24T20:45:22.338 INFO:tasks.ceph.ceph_manager.c2:waiting for all up 2021-03-24T20:45:22.338 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-03-24T20:45:22.603 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:22.604 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":14,"fsid":"19ff7635-a2a7-4347-a6a0-ab464f77d5fb","created":"2021-03-24T20:45:10.980569+0000","modified":"2021-03-24T20:45:22.114973+0000","last_up_change":"2021-03-24T20:45:16.069521+0000","last_in_change":"2021-03-24T20:45:16.069521+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-24T20:45:15.056543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-24T20:45:19.412470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"4716e8a9-b4a7-4c50-a4e7-8c860600febd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":12758},{"type":"v1","addr":"172.21.15.63:6817","nonce":12758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":12758},{"type":"v1","addr":"172.21.15.63:6819","nonce":12758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":12758},{"type":"v1","addr":"172.21.15.63:6823","nonce":12758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":12758},{"type":"v1","addr":"172.21.15.63:6821","nonce":12758}]},"public_addr":"172.21.15.63:6817/12758","cluster_addr":"172.21.15.63:6819/12758","heartbeat_back_addr":"172.21.15.63:6823/12758","heartbeat_front_addr":"172.21.15.63:6821/12758","state":["exists","up"]},{"osd":1,"uuid":"e72ecbde-5057-4f63-b5ce-ce1fe8f0cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":12759},{"type":"v1","addr":"172.21.15.63:6809","nonce":12759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":12759},{"type":"v1","addr":"172.21.15.63:6811","nonce":12759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":12759},{"type":"v1","addr":"172.21.15.63:6815","nonce":12759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":12759},{"type":"v1","addr":"172.21.15.63:6813","nonce":12759}]},"public_addr":"172.21.15.63:6809/12759","cluster_addr":"172.21.15.63:6811/12759","heartbeat_back_addr":"172.21.15.63:6815/12759","heartbeat_front_addr":"172.21.15.63:6813/12759","state":["exists","up"]},{"osd":2,"uuid":"d87e26ae-d46d-4786-b961-8369e589f488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":12757},{"type":"v1","addr":"172.21.15.63:6801","nonce":12757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":12757},{"type":"v1","addr":"172.21.15.63:6803","nonce":12757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":12757},{"type":"v1","addr":"172.21.15.63:6807","nonce":12757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":12757},{"type":"v1","addr":"172.21.15.63:6805","nonce":12757}]},"public_addr":"172.21.15.63:6801/12757","cluster_addr":"172.21.15.63:6803/12757","heartbeat_back_addr":"172.21.15.63:6807/12757","heartbeat_front_addr":"172.21.15.63:6805/12757","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:45:14.259698+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:45:13.993033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:45:13.945932+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-24T20:45:22.621 INFO:tasks.ceph.ceph_manager.c2:all up! 2021-03-24T20:45:22.622 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-03-24T20:45:22.892 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:22.892 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":14,"fsid":"19ff7635-a2a7-4347-a6a0-ab464f77d5fb","created":"2021-03-24T20:45:10.980569+0000","modified":"2021-03-24T20:45:22.114973+0000","last_up_change":"2021-03-24T20:45:16.069521+0000","last_in_change":"2021-03-24T20:45:16.069521+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-24T20:45:15.056543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-24T20:45:19.412470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"4716e8a9-b4a7-4c50-a4e7-8c860600febd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":12758},{"type":"v1","addr":"172.21.15.63:6817","nonce":12758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":12758},{"type":"v1","addr":"172.21.15.63:6819","nonce":12758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":12758},{"type":"v1","addr":"172.21.15.63:6823","nonce":12758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":12758},{"type":"v1","addr":"172.21.15.63:6821","nonce":12758}]},"public_addr":"172.21.15.63:6817/12758","cluster_addr":"172.21.15.63:6819/12758","heartbeat_back_addr":"172.21.15.63:6823/12758","heartbeat_front_addr":"172.21.15.63:6821/12758","state":["exists","up"]},{"osd":1,"uuid":"e72ecbde-5057-4f63-b5ce-ce1fe8f0cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":12759},{"type":"v1","addr":"172.21.15.63:6809","nonce":12759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":12759},{"type":"v1","addr":"172.21.15.63:6811","nonce":12759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":12759},{"type":"v1","addr":"172.21.15.63:6815","nonce":12759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":12759},{"type":"v1","addr":"172.21.15.63:6813","nonce":12759}]},"public_addr":"172.21.15.63:6809/12759","cluster_addr":"172.21.15.63:6811/12759","heartbeat_back_addr":"172.21.15.63:6815/12759","heartbeat_front_addr":"172.21.15.63:6813/12759","state":["exists","up"]},{"osd":2,"uuid":"d87e26ae-d46d-4786-b961-8369e589f488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":12757},{"type":"v1","addr":"172.21.15.63:6801","nonce":12757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":12757},{"type":"v1","addr":"172.21.15.63:6803","nonce":12757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":12757},{"type":"v1","addr":"172.21.15.63:6807","nonce":12757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":12757},{"type":"v1","addr":"172.21.15.63:6805","nonce":12757}]},"public_addr":"172.21.15.63:6801/12757","cluster_addr":"172.21.15.63:6803/12757","heartbeat_back_addr":"172.21.15.63:6807/12757","heartbeat_front_addr":"172.21.15.63:6805/12757","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:45:14.259698+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:45:13.993033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:45:13.945932+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-24T20:45:22.905 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.0 flush_pg_stats 2021-03-24T20:45:23.065 INFO:teuthology.orchestra.run.smithi063.stdout:38654705668 2021-03-24T20:45:23.066 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.1 flush_pg_stats 2021-03-24T20:45:23.229 INFO:teuthology.orchestra.run.smithi063.stdout:34359738371 2021-03-24T20:45:23.230 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 tell osd.2 flush_pg_stats 2021-03-24T20:45:23.395 INFO:teuthology.orchestra.run.smithi063.stdout:34359738371 2021-03-24T20:45:23.396 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-03-24T20:45:23.676 INFO:teuthology.orchestra.run.smithi063.stdout:38654705667 2021-03-24T20:45:23.690 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705667 for osd.0 2021-03-24T20:45:24.692 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-03-24T20:45:24.976 INFO:teuthology.orchestra.run.smithi063.stdout:38654705667 2021-03-24T20:45:24.989 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705667 for osd.0 2021-03-24T20:45:25.991 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.0 2021-03-24T20:45:26.273 INFO:teuthology.orchestra.run.smithi063.stdout:38654705668 2021-03-24T20:45:26.286 INFO:tasks.ceph.ceph_manager.c2:need seq 38654705668 got 38654705668 for osd.0 2021-03-24T20:45:26.287 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.1 2021-03-24T20:45:26.557 INFO:teuthology.orchestra.run.smithi063.stdout:34359738371 2021-03-24T20:45:26.570 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.1 2021-03-24T20:45:26.571 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd last-stat-seq osd.2 2021-03-24T20:45:26.844 INFO:teuthology.orchestra.run.smithi063.stdout:34359738371 2021-03-24T20:45:26.857 INFO:tasks.ceph.ceph_manager.c2:need seq 34359738371 got 34359738371 for osd.2 2021-03-24T20:45:26.858 INFO:tasks.ceph.ceph_manager.c2:waiting for clean 2021-03-24T20:45:26.858 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-03-24T20:45:27.125 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:27.128 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2021-03-24T20:45:27.139 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2021-03-24T20:45:26.970431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146100,"kb_used_data":348,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426700,"statfs":{"total":322122547200,"available":318900940800,"internally_reserved":3221225472,"allocated":356352,"data_stored":102967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.862542"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.125869+0000","last_change":"2021-03-24T20:45:21.121753+0000","last_active":"2021-03-24T20:45:21.125869+0000","last_peered":"2021-03-24T20:45:21.125869+0000","last_clean":"2021-03-24T20:45:21.125869+0000","last_became_active":"2021-03-24T20:45:21.121345+0000","last_became_peered":"2021-03-24T20:45:21.121345+0000","last_unstale":"2021-03-24T20:45:21.125869+0000","last_undegraded":"2021-03-24T20:45:21.125869+0000","last_fullsized":"2021-03-24T20:45:21.125869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_clean_scrub_stamp":"2021-03-24T20:45:20.098439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.128936+0000","last_change":"2021-03-24T20:45:21.124226+0000","last_active":"2021-03-24T20:45:21.128936+0000","last_peered":"2021-03-24T20:45:21.128936+0000","last_clean":"2021-03-24T20:45:21.128936+0000","last_became_active":"2021-03-24T20:45:21.123828+0000","last_became_peered":"2021-03-24T20:45:21.123828+0000","last_unstale":"2021-03-24T20:45:21.128936+0000","last_undegraded":"2021-03-24T20:45:21.128936+0000","last_fullsized":"2021-03-24T20:45:21.128936+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_clean_scrub_stamp":"2021-03-24T20:45:20.098439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.128950+0000","last_change":"2021-03-24T20:45:21.124249+0000","last_active":"2021-03-24T20:45:21.128950+0000","last_peered":"2021-03-24T20:45:21.128950+0000","last_clean":"2021-03-24T20:45:21.128950+0000","last_became_active":"2021-03-24T20:45:21.123903+0000","last_became_peered":"2021-03-24T20:45:21.123903+0000","last_unstale":"2021-03-24T20:45:21.128950+0000","last_undegraded":"2021-03-24T20:45:21.128950+0000","last_fullsized":"2021-03-24T20:45:21.128950+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_clean_scrub_stamp":"2021-03-24T20:45:20.098439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.128670+0000","last_change":"2021-03-24T20:45:21.124129+0000","last_active":"2021-03-24T20:45:21.128670+0000","last_peered":"2021-03-24T20:45:21.128670+0000","last_clean":"2021-03-24T20:45:21.128670+0000","last_became_active":"2021-03-24T20:45:21.123676+0000","last_became_peered":"2021-03-24T20:45:21.123676+0000","last_unstale":"2021-03-24T20:45:21.128670+0000","last_undegraded":"2021-03-24T20:45:21.128670+0000","last_fullsized":"2021-03-24T20:45:21.128670+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_clean_scrub_stamp":"2021-03-24T20:45:20.098439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'1","reported_seq":"15","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.110245+0000","last_change":"2021-03-24T20:45:17.092165+0000","last_active":"2021-03-24T20:45:21.110245+0000","last_peered":"2021-03-24T20:45:21.110245+0000","last_clean":"2021-03-24T20:45:21.110245+0000","last_became_active":"2021-03-24T20:45:17.090536+0000","last_became_peered":"2021-03-24T20:45:17.090536+0000","last_unstale":"2021-03-24T20:45:21.110245+0000","last_undegraded":"2021-03-24T20:45:21.110245+0000","last_fullsized":"2021-03-24T20:45:21.110245+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":8,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:15.056660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:15.056660+0000","last_clean_scrub_stamp":"2021-03-24T20:45:15.056660+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.126145+0000","last_change":"2021-03-24T20:45:21.122005+0000","last_active":"2021-03-24T20:45:21.126145+0000","last_peered":"2021-03-24T20:45:21.126145+0000","last_clean":"2021-03-24T20:45:21.126145+0000","last_became_active":"2021-03-24T20:45:21.121520+0000","last_became_peered":"2021-03-24T20:45:21.121520+0000","last_unstale":"2021-03-24T20:45:21.126145+0000","last_undegraded":"2021-03-24T20:45:21.126145+0000","last_fullsized":"2021-03-24T20:45:21.126145+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_clean_scrub_stamp":"2021-03-24T20:45:20.098439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.128659+0000","last_change":"2021-03-24T20:45:21.124088+0000","last_active":"2021-03-24T20:45:21.128659+0000","last_peered":"2021-03-24T20:45:21.128659+0000","last_clean":"2021-03-24T20:45:21.128659+0000","last_became_active":"2021-03-24T20:45:21.123648+0000","last_became_peered":"2021-03-24T20:45:21.123648+0000","last_unstale":"2021-03-24T20:45:21.128659+0000","last_undegraded":"2021-03-24T20:45:21.128659+0000","last_fullsized":"2021-03-24T20:45:21.128659+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_clean_scrub_stamp":"2021-03-24T20:45:20.098439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.122054+0000","last_change":"2021-03-24T20:45:21.122054+0000","last_active":"2021-03-24T20:45:21.122054+0000","last_peered":"2021-03-24T20:45:21.122054+0000","last_clean":"2021-03-24T20:45:21.122054+0000","last_became_active":"2021-03-24T20:45:21.121665+0000","last_became_peered":"2021-03-24T20:45:21.121665+0000","last_unstale":"2021-03-24T20:45:21.122054+0000","last_undegraded":"2021-03-24T20:45:21.122054+0000","last_fullsized":"2021-03-24T20:45:21.122054+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_clean_scrub_stamp":"2021-03-24T20:45:20.098439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.123162+0000","last_change":"2021-03-24T20:45:21.119771+0000","last_active":"2021-03-24T20:45:21.123162+0000","last_peered":"2021-03-24T20:45:21.123162+0000","last_clean":"2021-03-24T20:45:21.123162+0000","last_became_active":"2021-03-24T20:45:21.119180+0000","last_became_peered":"2021-03-24T20:45:21.119180+0000","last_unstale":"2021-03-24T20:45:21.123162+0000","last_undegraded":"2021-03-24T20:45:21.123162+0000","last_fullsized":"2021-03-24T20:45:21.123162+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_clean_scrub_stamp":"2021-03-24T20:45:20.098439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":32342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048704,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808896,"statfs":{"total":107374182400,"available":106300309504,"internally_reserved":1073741824,"allocated":122880,"data_stored":35855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048700,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808900,"statfs":{"total":107374182400,"available":106300313600,"internally_reserved":1073741824,"allocated":118784,"data_stored":34770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]}]}]}} 2021-03-24T20:45:27.140 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 pg dump --format=json 2021-03-24T20:45:27.416 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:27.418 INFO:teuthology.orchestra.run.smithi063.stderr:dumped all 2021-03-24T20:45:27.429 INFO:teuthology.orchestra.run.smithi063.stdout:{"pg_ready":true,"pg_map":{"version":16,"stamp":"2021-03-24T20:45:26.970431+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":314572800,"kb_used":3146100,"kb_used_data":348,"kb_used_omap":0,"kb_used_meta":3145728,"kb_avail":311426700,"statfs":{"total":322122547200,"available":318900940800,"internally_reserved":3221225472,"allocated":356352,"data_stored":102967,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":3221225472},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"5.862542"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.125869+0000","last_change":"2021-03-24T20:45:21.121753+0000","last_active":"2021-03-24T20:45:21.125869+0000","last_peered":"2021-03-24T20:45:21.125869+0000","last_clean":"2021-03-24T20:45:21.125869+0000","last_became_active":"2021-03-24T20:45:21.121345+0000","last_became_peered":"2021-03-24T20:45:21.121345+0000","last_unstale":"2021-03-24T20:45:21.125869+0000","last_undegraded":"2021-03-24T20:45:21.125869+0000","last_fullsized":"2021-03-24T20:45:21.125869+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_clean_scrub_stamp":"2021-03-24T20:45:20.098439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.128936+0000","last_change":"2021-03-24T20:45:21.124226+0000","last_active":"2021-03-24T20:45:21.128936+0000","last_peered":"2021-03-24T20:45:21.128936+0000","last_clean":"2021-03-24T20:45:21.128936+0000","last_became_active":"2021-03-24T20:45:21.123828+0000","last_became_peered":"2021-03-24T20:45:21.123828+0000","last_unstale":"2021-03-24T20:45:21.128936+0000","last_undegraded":"2021-03-24T20:45:21.128936+0000","last_fullsized":"2021-03-24T20:45:21.128936+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_clean_scrub_stamp":"2021-03-24T20:45:20.098439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.128950+0000","last_change":"2021-03-24T20:45:21.124249+0000","last_active":"2021-03-24T20:45:21.128950+0000","last_peered":"2021-03-24T20:45:21.128950+0000","last_clean":"2021-03-24T20:45:21.128950+0000","last_became_active":"2021-03-24T20:45:21.123903+0000","last_became_peered":"2021-03-24T20:45:21.123903+0000","last_unstale":"2021-03-24T20:45:21.128950+0000","last_undegraded":"2021-03-24T20:45:21.128950+0000","last_fullsized":"2021-03-24T20:45:21.128950+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_clean_scrub_stamp":"2021-03-24T20:45:20.098439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.128670+0000","last_change":"2021-03-24T20:45:21.124129+0000","last_active":"2021-03-24T20:45:21.128670+0000","last_peered":"2021-03-24T20:45:21.128670+0000","last_clean":"2021-03-24T20:45:21.128670+0000","last_became_active":"2021-03-24T20:45:21.123676+0000","last_became_peered":"2021-03-24T20:45:21.123676+0000","last_unstale":"2021-03-24T20:45:21.128670+0000","last_undegraded":"2021-03-24T20:45:21.128670+0000","last_fullsized":"2021-03-24T20:45:21.128670+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_clean_scrub_stamp":"2021-03-24T20:45:20.098439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"10'1","reported_seq":"15","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.110245+0000","last_change":"2021-03-24T20:45:17.092165+0000","last_active":"2021-03-24T20:45:21.110245+0000","last_peered":"2021-03-24T20:45:21.110245+0000","last_clean":"2021-03-24T20:45:21.110245+0000","last_became_active":"2021-03-24T20:45:17.090536+0000","last_became_peered":"2021-03-24T20:45:17.090536+0000","last_unstale":"2021-03-24T20:45:21.110245+0000","last_undegraded":"2021-03-24T20:45:21.110245+0000","last_fullsized":"2021-03-24T20:45:21.110245+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":8,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:15.056660+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:15.056660+0000","last_clean_scrub_stamp":"2021-03-24T20:45:15.056660+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.126145+0000","last_change":"2021-03-24T20:45:21.122005+0000","last_active":"2021-03-24T20:45:21.126145+0000","last_peered":"2021-03-24T20:45:21.126145+0000","last_clean":"2021-03-24T20:45:21.126145+0000","last_became_active":"2021-03-24T20:45:21.121520+0000","last_became_peered":"2021-03-24T20:45:21.121520+0000","last_unstale":"2021-03-24T20:45:21.126145+0000","last_undegraded":"2021-03-24T20:45:21.126145+0000","last_fullsized":"2021-03-24T20:45:21.126145+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_clean_scrub_stamp":"2021-03-24T20:45:20.098439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.128659+0000","last_change":"2021-03-24T20:45:21.124088+0000","last_active":"2021-03-24T20:45:21.128659+0000","last_peered":"2021-03-24T20:45:21.128659+0000","last_clean":"2021-03-24T20:45:21.128659+0000","last_became_active":"2021-03-24T20:45:21.123648+0000","last_became_peered":"2021-03-24T20:45:21.123648+0000","last_unstale":"2021-03-24T20:45:21.128659+0000","last_undegraded":"2021-03-24T20:45:21.128659+0000","last_fullsized":"2021-03-24T20:45:21.128659+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_clean_scrub_stamp":"2021-03-24T20:45:20.098439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":"8","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.122054+0000","last_change":"2021-03-24T20:45:21.122054+0000","last_active":"2021-03-24T20:45:21.122054+0000","last_peered":"2021-03-24T20:45:21.122054+0000","last_clean":"2021-03-24T20:45:21.122054+0000","last_became_active":"2021-03-24T20:45:21.121665+0000","last_became_peered":"2021-03-24T20:45:21.121665+0000","last_unstale":"2021-03-24T20:45:21.122054+0000","last_undegraded":"2021-03-24T20:45:21.122054+0000","last_fullsized":"2021-03-24T20:45:21.122054+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_clean_scrub_stamp":"2021-03-24T20:45:20.098439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":"9","reported_epoch":"13","state":"active+clean","last_fresh":"2021-03-24T20:45:21.123162+0000","last_change":"2021-03-24T20:45:21.119771+0000","last_active":"2021-03-24T20:45:21.123162+0000","last_peered":"2021-03-24T20:45:21.123162+0000","last_clean":"2021-03-24T20:45:21.123162+0000","last_became_active":"2021-03-24T20:45:21.119180+0000","last_became_peered":"2021-03-24T20:45:21.119180+0000","last_unstale":"2021-03-24T20:45:21.123162+0000","last_undegraded":"2021-03-24T20:45:21.123162+0000","last_fullsized":"2021-03-24T20:45:21.123162+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-03-24T20:45:20.098439+0000","last_clean_scrub_stamp":"2021-03-24T20:45:20.098439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":0,"up_from":9,"seq":38654705668,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048696,"kb_used_data":112,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808904,"statfs":{"total":107374182400,"available":106300317696,"internally_reserved":1073741824,"allocated":114688,"data_stored":32342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60599999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56899999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048704,"kb_used_data":120,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808896,"statfs":{"total":107374182400,"available":106300309504,"internally_reserved":1073741824,"allocated":122880,"data_stored":35855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.56399999999999995}]}]},{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":104857600,"kb_used":1048700,"kb_used_data":116,"kb_used_omap":0,"kb_used_meta":1048576,"kb_avail":103808900,"statfs":{"total":107374182400,"available":106300313600,"internally_reserved":1073741824,"allocated":118784,"data_stored":34770,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1073741824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32100000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.495}]}]}]}} 2021-03-24T20:45:27.430 INFO:tasks.ceph.ceph_manager.c2:clean! 2021-03-24T20:45:27.430 INFO:tasks.ceph:Waiting until ceph cluster c2 is healthy... 2021-03-24T20:45:27.431 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy 2021-03-24T20:45:27.431 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 health --format=json 2021-03-24T20:45:27.752 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T20:45:27.769 INFO:teuthology.orchestra.run.smithi063.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-03-24T20:45:27.770 INFO:tasks.ceph.ceph_manager.c2:wait_until_healthy done 2021-03-24T20:45:27.771 INFO:teuthology.run_tasks:Running task parallel... 2021-03-24T20:45:27.783 INFO:teuthology.task.parallel:starting parallel... 2021-03-24T20:45:27.784 INFO:teuthology.task.parallel:In parallel, running task sequential... 2021-03-24T20:45:27.786 INFO:teuthology.task.sequential:In sequential, running task install.upgrade... 2021-03-24T20:45:27.786 INFO:teuthology.task.install:project ceph config {'c2.client.0': None, 'c2.client.1': None, 'c2.mgr.x': None, 'c2.mon.a': None, 'c2.osd.0': None, 'c2.osd.1': None, 'c2.osd.2': None} overrides {'branch': 'octopus', 'sha1': '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2'} 2021-03-24T20:45:27.787 WARNING:teuthology.task.install:remote ubuntu@smithi063.front.sepia.ceph.com came up twice (role c2.client.1) 2021-03-24T20:45:27.788 WARNING:teuthology.task.install:remote ubuntu@smithi063.front.sepia.ceph.com came up twice (role c2.mgr.x) 2021-03-24T20:45:27.788 WARNING:teuthology.task.install:remote ubuntu@smithi063.front.sepia.ceph.com came up twice (role c2.mon.a) 2021-03-24T20:45:27.788 WARNING:teuthology.task.install:remote ubuntu@smithi063.front.sepia.ceph.com came up twice (role c2.osd.0) 2021-03-24T20:45:27.789 WARNING:teuthology.task.install:remote ubuntu@smithi063.front.sepia.ceph.com came up twice (role c2.osd.1) 2021-03-24T20:45:27.789 WARNING:teuthology.task.install:remote ubuntu@smithi063.front.sepia.ceph.com came up twice (role c2.osd.2) 2021-03-24T20:45:27.789 INFO:teuthology.task.install:remote ubuntu@smithi063.front.sepia.ceph.com config {'branch': 'octopus', 'sha1': '9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2'} 2021-03-24T20:45:27.790 INFO:teuthology.task.install:extra packages: [] 2021-03-24T20:45:27.790 DEBUG:teuthology.orchestra.run.smithi063:> sudo lsb_release -is 2021-03-24T20:45:27.843 INFO:teuthology.orchestra.run.smithi063.stdout:Ubuntu 2021-03-24T20:45:27.847 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-24T20:45:27.854 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-03-24T20:45:27.854 INFO:teuthology.task.install:Upgrading ceph deb packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse 2021-03-24T20:45:27.855 DEBUG:teuthology.orchestra.run.smithi063:> dpkg-query -W -f '${Version}' ceph-common 2021-03-24T20:45:27.878 INFO:teuthology.orchestra.run.smithi063.stdout:15.2.10-76-gdcf77bd9-1bionic 2021-03-24T20:45:27.879 INFO:teuthology.packaging:The installed version of ceph-common is 15.2.10-76-gdcf77bd9-1bionic 2021-03-24T20:45:27.879 DEBUG:teuthology.orchestra.run.smithi063:> sudo lsb_release -is 2021-03-24T20:45:27.972 INFO:teuthology.orchestra.run.smithi063.stdout:Ubuntu 2021-03-24T20:45:27.978 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-24T20:45:27.978 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-03-24T20:45:27.979 INFO:teuthology.packaging:ref: None 2021-03-24T20:45:27.980 INFO:teuthology.packaging:tag: None 2021-03-24T20:45:27.981 INFO:teuthology.packaging:branch: octopus 2021-03-24T20:45:27.981 INFO:teuthology.packaging:sha1: 9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2 2021-03-24T20:45:27.982 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=octopus 2021-03-24T20:45:28.240 INFO:teuthology.task.install:Ceph deb upgrade from 15.2.10-76-gdcf77bd9-1bionic to 15.2.10-76-gdcf77bd9-1bionic 2021-03-24T20:45:28.241 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-key list | grep Ceph 2021-03-24T20:45:28.343 INFO:teuthology.orchestra.run.smithi063.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2021-03-24T20:45:28.404 INFO:teuthology.orchestra.run.smithi063.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2021-03-24T20:45:28.404 INFO:teuthology.orchestra.run.smithi063.stdout:uid [ unknown] Ceph.com (release key) 2021-03-24T20:45:28.406 DEBUG:teuthology.orchestra.run.smithi063:> sudo lsb_release -is 2021-03-24T20:45:28.461 INFO:teuthology.orchestra.run.smithi063.stdout:Ubuntu 2021-03-24T20:45:28.467 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-24T20:45:28.467 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-03-24T20:45:28.468 INFO:teuthology.packaging:ref: None 2021-03-24T20:45:28.468 INFO:teuthology.packaging:tag: None 2021-03-24T20:45:28.469 INFO:teuthology.packaging:branch: octopus 2021-03-24T20:45:28.469 INFO:teuthology.packaging:sha1: 9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2 2021-03-24T20:45:28.469 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=octopus 2021-03-24T20:45:28.692 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default/ 2021-03-24T20:45:28.693 INFO:teuthology.task.install.deb:Package version is 15.2.10-76-gdcf77bd9-1bionic 2021-03-24T20:45:28.960 DEBUG:teuthology.orchestra.run.smithi063:> set -ex 2021-03-24T20:45:28.961 DEBUG:teuthology.orchestra.run.smithi063:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2021-03-24T20:45:28.976 DEBUG:teuthology.orchestra.run.smithi063:> sudo apt-get update 2021-03-24T20:45:29.161 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:1 http://security.ubuntu.com/ubuntu bionic-security InRelease 2021-03-24T20:45:29.276 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:2 http://archive.ubuntu.com/ubuntu bionic InRelease 2021-03-24T20:45:29.355 INFO:teuthology.orchestra.run.smithi063.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic InRelease 2021-03-24T20:45:29.356 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:4 http://archive.ubuntu.com/ubuntu bionic-updates InRelease 2021-03-24T20:45:29.411 INFO:teuthology.orchestra.run.smithi063.stdout:Hit:5 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic Release 2021-03-24T20:45:29.437 INFO:teuthology.orchestra.run.smithi063.stdout:Get:6 http://archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB] 2021-03-24T20:45:29.468 INFO:teuthology.orchestra.run.smithi063.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/octopus/dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5/ubuntu/bionic/flavors/default bionic Release.gpg 2021-03-24T20:45:29.810 INFO:teuthology.orchestra.run.smithi063.stdout:Fetched 74.6 kB in 1s (119 kB/s) 2021-03-24T20:45:31.126 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2021-03-24T20:45:31.174 DEBUG:teuthology.orchestra.run.smithi063:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.10-76-gdcf77bd9-1bionic cephadm=15.2.10-76-gdcf77bd9-1bionic ceph-mds=15.2.10-76-gdcf77bd9-1bionic ceph-mgr=15.2.10-76-gdcf77bd9-1bionic ceph-common=15.2.10-76-gdcf77bd9-1bionic ceph-fuse=15.2.10-76-gdcf77bd9-1bionic ceph-test=15.2.10-76-gdcf77bd9-1bionic radosgw=15.2.10-76-gdcf77bd9-1bionic python3-rados=15.2.10-76-gdcf77bd9-1bionic python3-rgw=15.2.10-76-gdcf77bd9-1bionic python3-cephfs=15.2.10-76-gdcf77bd9-1bionic python3-rbd=15.2.10-76-gdcf77bd9-1bionic libcephfs2=15.2.10-76-gdcf77bd9-1bionic libcephfs-dev=15.2.10-76-gdcf77bd9-1bionic librados2=15.2.10-76-gdcf77bd9-1bionic librbd1=15.2.10-76-gdcf77bd9-1bionic rbd-fuse=15.2.10-76-gdcf77bd9-1bionic 2021-03-24T20:45:31.259 INFO:teuthology.orchestra.run.smithi063.stdout:Reading package lists... 2021-03-24T20:45:31.528 INFO:teuthology.orchestra.run.smithi063.stdout:Building dependency tree... 2021-03-24T20:45:31.529 INFO:teuthology.orchestra.run.smithi063.stdout:Reading state information... 2021-03-24T20:45:31.752 INFO:teuthology.orchestra.run.smithi063.stdout:ceph is already the newest version (15.2.10-76-gdcf77bd9-1bionic). 2021-03-24T20:45:31.753 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-common is already the newest version (15.2.10-76-gdcf77bd9-1bionic). 2021-03-24T20:45:31.753 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-fuse is already the newest version (15.2.10-76-gdcf77bd9-1bionic). 2021-03-24T20:45:31.753 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-mds is already the newest version (15.2.10-76-gdcf77bd9-1bionic). 2021-03-24T20:45:31.753 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-mgr is already the newest version (15.2.10-76-gdcf77bd9-1bionic). 2021-03-24T20:45:31.754 INFO:teuthology.orchestra.run.smithi063.stdout:ceph-test is already the newest version (15.2.10-76-gdcf77bd9-1bionic). 2021-03-24T20:45:31.754 INFO:teuthology.orchestra.run.smithi063.stdout:cephadm is already the newest version (15.2.10-76-gdcf77bd9-1bionic). 2021-03-24T20:45:31.754 INFO:teuthology.orchestra.run.smithi063.stdout:libcephfs-dev is already the newest version (15.2.10-76-gdcf77bd9-1bionic). 2021-03-24T20:45:31.755 INFO:teuthology.orchestra.run.smithi063.stdout:libcephfs2 is already the newest version (15.2.10-76-gdcf77bd9-1bionic). 2021-03-24T20:45:31.755 INFO:teuthology.orchestra.run.smithi063.stdout:librados2 is already the newest version (15.2.10-76-gdcf77bd9-1bionic). 2021-03-24T20:45:31.755 INFO:teuthology.orchestra.run.smithi063.stdout:librados2 set to manually installed. 2021-03-24T20:45:31.755 INFO:teuthology.orchestra.run.smithi063.stdout:librbd1 is already the newest version (15.2.10-76-gdcf77bd9-1bionic). 2021-03-24T20:45:31.756 INFO:teuthology.orchestra.run.smithi063.stdout:librbd1 set to manually installed. 2021-03-24T20:45:31.756 INFO:teuthology.orchestra.run.smithi063.stdout:python3-cephfs is already the newest version (15.2.10-76-gdcf77bd9-1bionic). 2021-03-24T20:45:31.756 INFO:teuthology.orchestra.run.smithi063.stdout:python3-rados is already the newest version (15.2.10-76-gdcf77bd9-1bionic). 2021-03-24T20:45:31.756 INFO:teuthology.orchestra.run.smithi063.stdout:python3-rbd is already the newest version (15.2.10-76-gdcf77bd9-1bionic). 2021-03-24T20:45:31.757 INFO:teuthology.orchestra.run.smithi063.stdout:python3-rgw is already the newest version (15.2.10-76-gdcf77bd9-1bionic). 2021-03-24T20:45:31.757 INFO:teuthology.orchestra.run.smithi063.stdout:radosgw is already the newest version (15.2.10-76-gdcf77bd9-1bionic). 2021-03-24T20:45:31.757 INFO:teuthology.orchestra.run.smithi063.stdout:rbd-fuse is already the newest version (15.2.10-76-gdcf77bd9-1bionic). 2021-03-24T20:45:31.757 INFO:teuthology.orchestra.run.smithi063.stdout:0 upgraded, 0 newly installed, 0 to remove and 29 not upgraded. 2021-03-24T20:45:31.758 INFO:teuthology.orchestra.run.smithi063.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2021-03-24T20:45:31.759 DEBUG:teuthology.orchestra.run.smithi063:> sudo lsb_release -is 2021-03-24T20:45:31.808 INFO:teuthology.orchestra.run.smithi063.stdout:Ubuntu 2021-03-24T20:45:31.813 DEBUG:teuthology.misc:System to be installed: Ubuntu 2021-03-24T20:45:31.813 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-03-24T20:45:31.814 INFO:teuthology.packaging:ref: None 2021-03-24T20:45:31.814 INFO:teuthology.packaging:tag: None 2021-03-24T20:45:31.815 INFO:teuthology.packaging:branch: octopus 2021-03-24T20:45:31.815 INFO:teuthology.packaging:sha1: 9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2 2021-03-24T20:45:31.815 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F18.04%2Fx86_64&ref=octopus 2021-03-24T20:45:32.034 DEBUG:teuthology.orchestra.run.smithi063:> dpkg-query -W -f '${Version}' ceph 2021-03-24T20:45:32.058 INFO:teuthology.orchestra.run.smithi063.stdout:15.2.10-76-gdcf77bd9-1bionic 2021-03-24T20:45:32.059 INFO:teuthology.packaging:The installed version of ceph is 15.2.10-76-gdcf77bd9-1bionic 2021-03-24T20:45:32.059 INFO:teuthology.task.install:The correct ceph version 15.2.10-76-gdcf77bd9-1bionic is installed. 2021-03-24T20:45:32.060 DEBUG:teuthology.parallel:result is None 2021-03-24T20:45:32.060 INFO:teuthology.run_tasks:Running task rgw... 2021-03-24T20:45:32.082 DEBUG:tasks.rgw:config is {'c1.client.0': {'port': 8000}, 'c2.client.0': {'port': 8000}, 'c2.client.1': {'port': 8001}} 2021-03-24T20:45:32.082 DEBUG:tasks.rgw:client list is dict_keys(['c1.client.0', 'c2.client.0', 'c2.client.1']) 2021-03-24T20:45:32.083 INFO:tasks.rgw:Creating data pools 2021-03-24T20:45:32.083 DEBUG:tasks.rgw:Obtaining remote for client c1.client.0 2021-03-24T20:45:32.083 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c1 2021-03-24T20:45:32.509 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'default.rgw.buckets.data' created 2021-03-24T20:45:32.520 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c1 2021-03-24T20:45:33.526 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-03-24T20:45:33.543 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c1 2021-03-24T20:45:34.521 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'default.rgw.buckets.index' created 2021-03-24T20:45:34.532 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c1 2021-03-24T20:45:35.543 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-03-24T20:45:35.561 DEBUG:tasks.rgw:Obtaining remote for client c2.client.0 2021-03-24T20:45:35.561 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-03-24T20:45:36.038 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'default.rgw.buckets.data' created 2021-03-24T20:45:36.058 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-03-24T20:45:37.068 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-03-24T20:45:37.084 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-03-24T20:45:38.064 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'default.rgw.buckets.index' created 2021-03-24T20:45:38.079 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-03-24T20:45:39.092 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-03-24T20:45:39.110 DEBUG:tasks.rgw:Obtaining remote for client c2.client.1 2021-03-24T20:45:39.111 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool create default.rgw.buckets.data 64 64 --cluster c2 2021-03-24T20:45:39.384 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'default.rgw.buckets.data' already exists 2021-03-24T20:45:39.401 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool application enable default.rgw.buckets.data rgw --cluster c2 2021-03-24T20:45:40.083 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.data' 2021-03-24T20:45:40.095 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool create default.rgw.buckets.index 64 64 --cluster c2 2021-03-24T20:45:40.380 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'default.rgw.buckets.index' already exists 2021-03-24T20:45:40.390 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool application enable default.rgw.buckets.index rgw --cluster c2 2021-03-24T20:45:41.092 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rgw' on pool 'default.rgw.buckets.index' 2021-03-24T20:45:41.106 DEBUG:tasks.rgw:Pools created 2021-03-24T20:45:41.106 INFO:tasks.rgw:Configuring compression type = random 2021-03-24T20:45:41.106 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'list'] 2021-03-24T20:45:41.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'list'] 2021-03-24T20:45:41.108 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user list 2021-03-24T20:45:41.144 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:45:41.145 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:45:41.156 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:41.156+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:42.367 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.368+0000 7f2495c3a080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:42.368 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.368+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:42.376 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.376+0000 7f2495c3a080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:42.377 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.376+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:42.377 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.376+0000 7f2495c3a080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:42.378 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.376+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:42.378 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.376+0000 7f2495c3a080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:42.393 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.392+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:42.394 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.392+0000 7f2495c3a080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:42.394 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.392+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:42.395 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.392+0000 7f2495c3a080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:42.395 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.392+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:42.396 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.392+0000 7f2495c3a080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:42.396 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.392+0000 7f2495c3a080 10 cannot find current period zonegroup using local zonegroup 2021-03-24T20:45:42.396 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.392+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:42.397 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.392+0000 7f2495c3a080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:42.397 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.392+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:42.398 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.392+0000 7f2495c3a080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:42.398 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.392+0000 7f2495c3a080 10 Creating default zonegroup 2021-03-24T20:45:42.399 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.392+0000 7f2495c3a080 20 get_system_obj_state: rctx=0x7ffc23c6e190 obj=:.pools.avail state=0x565057c9aec0 s->prefetch_data=0 2021-03-24T20:45:42.400 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.392+0000 7f2495c3a080 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-03-24T20:45:42.405 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.404+0000 7f2495c3a080 10 WARNING: store->list_zones() returned r=-2 2021-03-24T20:45:42.405 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.404+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:42.405 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.404+0000 7f2495c3a080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:42.408 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.408+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:42.409 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.408+0000 7f2495c3a080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:42.409 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.408+0000 7f2495c3a080 10 could not read realm id: (2) No such file or directory 2021-03-24T20:45:42.410 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.408+0000 7f2495c3a080 10 WARNING: failed to set zone as default, r=-22 2021-03-24T20:45:42.410 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.408+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:42.410 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.408+0000 7f2495c3a080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:42.413 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.412+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:42.413 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.416+0000 7f2495c3a080 20 rados_obj.operate() r=0 bl.length=839 2021-03-24T20:45:42.414 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.416+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:42.415 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.416+0000 7f2495c3a080 20 rados_obj.operate() r=0 bl.length=358 2021-03-24T20:45:42.415 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.416+0000 7f2495c3a080 20 zonegroup default 2021-03-24T20:45:42.415 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.416+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:42.416 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.416+0000 7f2495c3a080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:42.416 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.416+0000 7f2495c3a080 10 Cannot find current period zone using local zone 2021-03-24T20:45:42.416 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.416+0000 7f2495c3a080 10 Using default name default 2021-03-24T20:45:42.416 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.416+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:42.417 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.416+0000 7f2495c3a080 20 rados_obj.operate() r=0 bl.length=46 2021-03-24T20:45:42.418 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.416+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:42.418 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.416+0000 7f2495c3a080 20 rados_obj.operate() r=0 bl.length=839 2021-03-24T20:45:42.418 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.416+0000 7f2495c3a080 20 zone default found 2021-03-24T20:45:42.418 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:42.416+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:44.395 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:44.396+0000 7f2495c3a080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:44.396 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:44.396+0000 7f2495c3a080 20 rados->read ofs=0 len=0 2021-03-24T20:45:44.398 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:44.400+0000 7f2495c3a080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:44.399 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:44.400+0000 7f2495c3a080 20 started sync module instance, tier type = 2021-03-24T20:45:44.399 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:44.400+0000 7f2495c3a080 20 started zone id=6171d264-2d77-4b6e-b661-344f908bf923 (name=default) with tier type = 2021-03-24T20:45:46.434 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:46.435+0000 7f2495c3a080 20 add_watcher() i=0 2021-03-24T20:45:46.435 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:46.435+0000 7f2495c3a080 20 add_watcher() i=1 2021-03-24T20:45:46.435 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:46.435+0000 7f2495c3a080 20 add_watcher() i=2 2021-03-24T20:45:46.435 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:46.435+0000 7f2495c3a080 20 add_watcher() i=3 2021-03-24T20:45:46.436 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:46.435+0000 7f2495c3a080 20 add_watcher() i=4 2021-03-24T20:45:46.436 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:46.435+0000 7f2495c3a080 20 add_watcher() i=5 2021-03-24T20:45:46.436 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:46.435+0000 7f2495c3a080 20 add_watcher() i=6 2021-03-24T20:45:46.437 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:46.435+0000 7f2495c3a080 20 add_watcher() i=7 2021-03-24T20:45:46.437 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:46.435+0000 7f2495c3a080 2 all 8 watchers are set, enabling cache 2021-03-24T20:45:46.438 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:46.435+0000 7f2495c3a080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-24T20:45:46.438 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:46.435+0000 7f2495c3a080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-24T20:45:46.438 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:46.435+0000 7f2451feb700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-24T20:45:50.421 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.417+0000 7f2495c3a080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-03-24T20:45:50.511 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.505+0000 7f23b7fff700 20 reqs_thread_entry: start 2021-03-24T20:45:50.512 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.509+0000 7f2495c3a080 20 init_complete bucket index max shards: 11 2021-03-24T20:45:50.512 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.509+0000 7f23b5ffb700 20 reqs_thread_entry: start 2021-03-24T20:45:50.512 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.509+0000 7f23a3fff700 20 reqs_thread_entry: start 2021-03-24T20:45:50.533 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.529+0000 7f2451feb700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-24T20:45:50.535 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.529+0000 7f2495c3a080 20 remove_watcher() i=0 2021-03-24T20:45:50.535 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.529+0000 7f2495c3a080 2 removed watcher, disabling cache 2021-03-24T20:45:50.537 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.533+0000 7f2495c3a080 20 remove_watcher() i=1 2021-03-24T20:45:50.539 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.533+0000 7f2495c3a080 20 remove_watcher() i=2 2021-03-24T20:45:50.541 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.537+0000 7f2495c3a080 20 remove_watcher() i=3 2021-03-24T20:45:50.543 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.541+0000 7f2495c3a080 20 remove_watcher() i=4 2021-03-24T20:45:50.546 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.541+0000 7f2495c3a080 20 remove_watcher() i=5 2021-03-24T20:45:50.548 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.545+0000 7f2495c3a080 20 remove_watcher() i=6 2021-03-24T20:45:50.550 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.545+0000 7f2495c3a080 20 remove_watcher() i=7 2021-03-24T20:45:50.558 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2021-03-24T20:45:50.559 DEBUG:tasks.util.rgw: json result: [] 2021-03-24T20:45:50.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-24T20:45:50.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-24T20:45:50.562 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-03-24T20:45:50.639 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:45:50.640 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:45:50.654 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.649+0000 7f2ff9883080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-24T20:45:50.654 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.649+0000 7f2ff9883080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-24T20:45:50.661 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.657+0000 7f2f2dffb700 20 reqs_thread_entry: start 2021-03-24T20:45:50.661 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.657+0000 7f2ff9883080 20 rados->read ofs=0 len=0 2021-03-24T20:45:50.663 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.657+0000 7f2ff9883080 20 rados_obj.operate() r=0 bl.length=46 2021-03-24T20:45:50.663 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.657+0000 7f2ff9883080 20 rados->read ofs=0 len=0 2021-03-24T20:45:50.664 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.661+0000 7f2ff9883080 20 rados_obj.operate() r=0 bl.length=839 2021-03-24T20:45:50.664 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.661+0000 7f2ff9883080 20 rados->read ofs=0 len=0 2021-03-24T20:45:50.665 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.661+0000 7f2ff9883080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:50.665 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.661+0000 7f2ff9883080 20 rados->read ofs=0 len=0 2021-03-24T20:45:50.666 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.661+0000 7f2ff9883080 20 rados_obj.operate() r=0 bl.length=46 2021-03-24T20:45:50.666 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.661+0000 7f2ff9883080 20 rados->read ofs=0 len=0 2021-03-24T20:45:50.666 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:45:50.661+0000 7f2ff9883080 20 rados_obj.operate() r=0 bl.length=358 2021-03-24T20:45:50.676 INFO:teuthology.orchestra.run.smithi052.stdout:{"id":"6171d264-2d77-4b6e-b661-344f908bf923","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-24T20:45:50.677 DEBUG:tasks.util.rgw: json result: {'id': '6171d264-2d77-4b6e-b661-344f908bf923', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2021-03-24T20:45:50.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['user', 'list'] 2021-03-24T20:45:50.678 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'user', 'list'] 2021-03-24T20:45:50.678 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 user list 2021-03-24T20:45:50.715 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:45:50.715 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:45:50.728 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:50.726+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:52.135 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.133+0000 7f68b274d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:52.136 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.133+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:52.141 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.141+0000 7f68b274d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:52.142 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.141+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:52.142 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.141+0000 7f68b274d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:52.142 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.141+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:52.143 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.141+0000 7f68b274d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:52.156 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.153+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:52.156 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.153+0000 7f68b274d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:52.157 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.153+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:52.157 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.153+0000 7f68b274d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:52.157 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.153+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:52.157 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.153+0000 7f68b274d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:52.157 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.153+0000 7f68b274d080 10 cannot find current period zonegroup using local zonegroup 2021-03-24T20:45:52.158 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.153+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:52.159 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.153+0000 7f68b274d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:52.159 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.153+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:52.159 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.153+0000 7f68b274d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:52.159 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.153+0000 7f68b274d080 10 Creating default zonegroup 2021-03-24T20:45:52.160 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.153+0000 7f68b274d080 20 get_system_obj_state: rctx=0x7ffd8c04b840 obj=:.pools.avail state=0x561f4d393ec0 s->prefetch_data=0 2021-03-24T20:45:52.160 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.157+0000 7f68b274d080 10 couldn't find old data placement pools config, setting up new ones for the zone 2021-03-24T20:45:52.168 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.165+0000 7f68b274d080 10 WARNING: store->list_zones() returned r=-2 2021-03-24T20:45:52.168 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.165+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:52.168 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.165+0000 7f68b274d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:52.173 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.169+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:52.174 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.169+0000 7f68b274d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:52.175 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.169+0000 7f68b274d080 10 could not read realm id: (2) No such file or directory 2021-03-24T20:45:52.175 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.169+0000 7f68b274d080 10 WARNING: failed to set zone as default, r=-22 2021-03-24T20:45:52.175 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.169+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:52.175 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.169+0000 7f68b274d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:52.176 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.173+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:52.177 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.173+0000 7f68b274d080 20 rados_obj.operate() r=0 bl.length=839 2021-03-24T20:45:52.177 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.173+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:52.178 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.177+0000 7f68b274d080 20 rados_obj.operate() r=0 bl.length=358 2021-03-24T20:45:52.179 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.177+0000 7f68b274d080 20 zonegroup default 2021-03-24T20:45:52.179 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.177+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:52.179 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.177+0000 7f68b274d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:52.179 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.177+0000 7f68b274d080 10 Cannot find current period zone using local zone 2021-03-24T20:45:52.180 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.177+0000 7f68b274d080 10 Using default name default 2021-03-24T20:45:52.180 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.177+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:52.180 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.177+0000 7f68b274d080 20 rados_obj.operate() r=0 bl.length=46 2021-03-24T20:45:52.180 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.177+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:52.181 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.177+0000 7f68b274d080 20 rados_obj.operate() r=0 bl.length=839 2021-03-24T20:45:52.181 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.177+0000 7f68b274d080 20 zone default found 2021-03-24T20:45:52.182 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:52.177+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:54.165 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:54.161+0000 7f68b274d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:54.166 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:54.161+0000 7f68b274d080 20 rados->read ofs=0 len=0 2021-03-24T20:45:54.167 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:54.165+0000 7f68b274d080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:45:54.167 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:54.165+0000 7f68b274d080 20 started sync module instance, tier type = 2021-03-24T20:45:54.167 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:54.165+0000 7f68b274d080 20 started zone id=ae4b6791-a8a5-4095-9b94-989035cfb1b5 (name=default) with tier type = 2021-03-24T20:45:56.203 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:56.201+0000 7f68b274d080 20 add_watcher() i=0 2021-03-24T20:45:56.203 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:56.201+0000 7f68b274d080 20 add_watcher() i=1 2021-03-24T20:45:56.204 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:56.201+0000 7f68b274d080 20 add_watcher() i=2 2021-03-24T20:45:56.204 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:56.201+0000 7f68b274d080 20 add_watcher() i=3 2021-03-24T20:45:56.205 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:56.201+0000 7f68b274d080 20 add_watcher() i=4 2021-03-24T20:45:56.205 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:56.201+0000 7f68b274d080 20 add_watcher() i=5 2021-03-24T20:45:56.205 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:56.201+0000 7f68b274d080 20 add_watcher() i=6 2021-03-24T20:45:56.207 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:56.201+0000 7f68b274d080 20 add_watcher() i=7 2021-03-24T20:45:56.207 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:56.201+0000 7f68b274d080 2 all 8 watchers are set, enabling cache 2021-03-24T20:45:56.208 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:56.201+0000 7f68b274d080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-24T20:45:56.208 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:56.201+0000 7f68b274d080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-24T20:45:56.208 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:45:56.201+0000 7f686e7ec700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-24T20:46:00.188 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.185+0000 7f68b274d080 10 rgw_init_ioctx warning: failed to set recovery_priority on default.rgw.meta 2021-03-24T20:46:00.411 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.409+0000 7f67e4ff9700 20 reqs_thread_entry: start 2021-03-24T20:46:00.412 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.409+0000 7f68b274d080 20 init_complete bucket index max shards: 11 2021-03-24T20:46:00.412 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.409+0000 7f67d27fc700 20 reqs_thread_entry: start 2021-03-24T20:46:00.412 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.409+0000 7f67d0ff9700 20 reqs_thread_entry: start 2021-03-24T20:46:00.434 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.429+0000 7f686e7ec700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-24T20:46:00.435 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.433+0000 7f68b274d080 20 remove_watcher() i=0 2021-03-24T20:46:00.436 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.433+0000 7f68b274d080 2 removed watcher, disabling cache 2021-03-24T20:46:00.437 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.433+0000 7f68b274d080 20 remove_watcher() i=1 2021-03-24T20:46:00.439 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.437+0000 7f68b274d080 20 remove_watcher() i=2 2021-03-24T20:46:00.441 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.437+0000 7f68b274d080 20 remove_watcher() i=3 2021-03-24T20:46:00.443 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.441+0000 7f68b274d080 20 remove_watcher() i=4 2021-03-24T20:46:00.445 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.441+0000 7f68b274d080 20 remove_watcher() i=5 2021-03-24T20:46:00.447 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.445+0000 7f68b274d080 20 remove_watcher() i=6 2021-03-24T20:46:00.449 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.445+0000 7f68b274d080 20 remove_watcher() i=7 2021-03-24T20:46:00.457 INFO:teuthology.orchestra.run.smithi063.stdout:[] 2021-03-24T20:46:00.458 DEBUG:tasks.util.rgw: json result: [] 2021-03-24T20:46:00.460 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-24T20:46:00.461 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-24T20:46:00.461 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-03-24T20:46:00.540 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:46:00.541 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:46:00.552 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.549+0000 7f10234af080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-24T20:46:00.552 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.549+0000 7f10234af080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-24T20:46:00.561 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.557+0000 7f0f56ffd700 20 reqs_thread_entry: start 2021-03-24T20:46:00.561 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.557+0000 7f10234af080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.562 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.561+0000 7f10234af080 20 rados_obj.operate() r=0 bl.length=46 2021-03-24T20:46:00.563 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.561+0000 7f10234af080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.563 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.561+0000 7f10234af080 20 rados_obj.operate() r=0 bl.length=839 2021-03-24T20:46:00.563 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.561+0000 7f10234af080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.564 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.561+0000 7f10234af080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:46:00.564 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.561+0000 7f10234af080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.565 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.561+0000 7f10234af080 20 rados_obj.operate() r=0 bl.length=46 2021-03-24T20:46:00.565 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.561+0000 7f10234af080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.565 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.561+0000 7f10234af080 20 rados_obj.operate() r=0 bl.length=358 2021-03-24T20:46:00.574 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"ae4b6791-a8a5-4095-9b94-989035cfb1b5","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-24T20:46:00.574 DEBUG:tasks.util.rgw: json result: {'id': 'ae4b6791-a8a5-4095-9b94-989035cfb1b5', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2021-03-24T20:46:00.575 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['user', 'list'] 2021-03-24T20:46:00.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'user', 'list'] 2021-03-24T20:46:00.576 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 user list 2021-03-24T20:46:00.654 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:46:00.654 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:46:00.667 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.665+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.669 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.665+0000 7fc53d716080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:46:00.670 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.665+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.670 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.665+0000 7fc53d716080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:46:00.671 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.665+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.671 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.665+0000 7fc53d716080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:46:00.672 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.665+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.673 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.665+0000 7fc53d716080 20 rados_obj.operate() r=0 bl.length=46 2021-03-24T20:46:00.691 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.689+0000 7fc53d716080 20 RGWRados::pool_iterate: got zonegroup_info.e77c4d69-4954-4fc8-992a-2bd3657b6af0 2021-03-24T20:46:00.692 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.689+0000 7fc53d716080 20 RGWRados::pool_iterate: got zone_info.ae4b6791-a8a5-4095-9b94-989035cfb1b5 2021-03-24T20:46:00.692 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.689+0000 7fc53d716080 20 RGWRados::pool_iterate: got zone_names.default 2021-03-24T20:46:00.693 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.689+0000 7fc53d716080 20 RGWRados::pool_iterate: got zonegroups_names.default 2021-03-24T20:46:00.693 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.689+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.693 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.689+0000 7fc53d716080 20 rados_obj.operate() r=0 bl.length=46 2021-03-24T20:46:00.693 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.689+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.696 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.689+0000 7fc53d716080 20 rados_obj.operate() r=0 bl.length=855 2021-03-24T20:46:00.696 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.689+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.697 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.689+0000 7fc53d716080 20 rados_obj.operate() r=0 bl.length=46 2021-03-24T20:46:00.697 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.689+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.698 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.693+0000 7fc53d716080 20 rados_obj.operate() r=0 bl.length=358 2021-03-24T20:46:00.698 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.693+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.698 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.693+0000 7fc53d716080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:46:00.699 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.693+0000 7fc53d716080 10 cannot find current period zonegroup using local zonegroup 2021-03-24T20:46:00.699 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.693+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.700 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.693+0000 7fc53d716080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:46:00.700 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.693+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.700 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.693+0000 7fc53d716080 20 rados_obj.operate() r=0 bl.length=46 2021-03-24T20:46:00.701 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.693+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.701 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.693+0000 7fc53d716080 20 rados_obj.operate() r=0 bl.length=358 2021-03-24T20:46:00.702 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.693+0000 7fc53d716080 20 zonegroup default 2021-03-24T20:46:00.702 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.693+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.706 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.693+0000 7fc53d716080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:46:00.706 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.693+0000 7fc53d716080 10 Cannot find current period zone using local zone 2021-03-24T20:46:00.706 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.693+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.707 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.693+0000 7fc53d716080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:46:00.707 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.693+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.707 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.697+0000 7fc53d716080 20 rados_obj.operate() r=0 bl.length=46 2021-03-24T20:46:00.707 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.697+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.708 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.697+0000 7fc53d716080 20 rados_obj.operate() r=0 bl.length=855 2021-03-24T20:46:00.708 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.697+0000 7fc53d716080 20 zone default found 2021-03-24T20:46:00.708 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.697+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.708 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.697+0000 7fc53d716080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:46:00.708 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.697+0000 7fc53d716080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.709 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.697+0000 7fc53d716080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:46:00.709 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.697+0000 7fc53d716080 20 started sync module instance, tier type = 2021-03-24T20:46:00.709 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.697+0000 7fc53d716080 20 started zone id=ae4b6791-a8a5-4095-9b94-989035cfb1b5 (name=default) with tier type = 2021-03-24T20:46:00.720 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.717+0000 7fc53d716080 20 add_watcher() i=0 2021-03-24T20:46:00.721 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.717+0000 7fc53d716080 20 add_watcher() i=1 2021-03-24T20:46:00.721 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.717+0000 7fc53d716080 20 add_watcher() i=2 2021-03-24T20:46:00.721 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.717+0000 7fc53d716080 20 add_watcher() i=3 2021-03-24T20:46:00.721 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.717+0000 7fc53d716080 20 add_watcher() i=4 2021-03-24T20:46:00.722 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.717+0000 7fc53d716080 20 add_watcher() i=5 2021-03-24T20:46:00.722 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.717+0000 7fc53d716080 20 add_watcher() i=6 2021-03-24T20:46:00.723 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.717+0000 7fc53d716080 20 add_watcher() i=7 2021-03-24T20:46:00.723 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.717+0000 7fc53d716080 2 all 8 watchers are set, enabling cache 2021-03-24T20:46:00.723 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.717+0000 7fc53d716080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-24T20:46:00.723 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.717+0000 7fc53d716080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-24T20:46:00.724 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.717+0000 7fc4f9fe3700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-24T20:46:00.805 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.801+0000 7fc46bfff700 20 reqs_thread_entry: start 2021-03-24T20:46:00.805 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.801+0000 7fc53d716080 20 init_complete bucket index max shards: 11 2021-03-24T20:46:00.806 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.801+0000 7fc469ffb700 20 reqs_thread_entry: start 2021-03-24T20:46:00.806 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.805+0000 7fc457fff700 20 reqs_thread_entry: start 2021-03-24T20:46:00.827 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.825+0000 7fc4f9fe3700 2 RGWDataChangesLog::ChangesRenewThread: start 2021-03-24T20:46:00.829 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.825+0000 7fc53d716080 20 remove_watcher() i=0 2021-03-24T20:46:00.829 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.825+0000 7fc53d716080 2 removed watcher, disabling cache 2021-03-24T20:46:00.832 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.829+0000 7fc53d716080 20 remove_watcher() i=1 2021-03-24T20:46:00.834 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.833+0000 7fc53d716080 20 remove_watcher() i=2 2021-03-24T20:46:00.838 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.833+0000 7fc53d716080 20 remove_watcher() i=3 2021-03-24T20:46:00.839 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.837+0000 7fc53d716080 20 remove_watcher() i=4 2021-03-24T20:46:00.842 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.841+0000 7fc53d716080 20 remove_watcher() i=5 2021-03-24T20:46:00.844 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.841+0000 7fc53d716080 20 remove_watcher() i=6 2021-03-24T20:46:00.846 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.845+0000 7fc53d716080 20 remove_watcher() i=7 2021-03-24T20:46:00.855 INFO:teuthology.orchestra.run.smithi063.stdout:[] 2021-03-24T20:46:00.855 DEBUG:tasks.util.rgw: json result: [] 2021-03-24T20:46:00.856 INFO:tasks.util.rgw:rgwadmin: c2.client.1 : ['zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-24T20:46:00.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.1', '--cluster', 'c2', 'zone', 'placement', 'modify', '--rgw-zone', 'default', '--placement-id', 'default-placement', '--compression', 'random'] 2021-03-24T20:46:00.856 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.1 --cluster c2 zone placement modify --rgw-zone default --placement-id default-placement --compression random 2021-03-24T20:46:00.940 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:46:00.941 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:46:00.953 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.949+0000 7fef6b705080 20 check_secure_mon_conn(): auth registy supported: methods=[2] modes=[2,1] 2021-03-24T20:46:00.953 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.949+0000 7fef6b705080 20 check_secure_mon_conn(): mode 1 is insecure 2021-03-24T20:46:00.963 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.961+0000 7fee9f7fe700 20 reqs_thread_entry: start 2021-03-24T20:46:00.963 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.961+0000 7fef6b705080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.964 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.961+0000 7fef6b705080 20 rados_obj.operate() r=0 bl.length=46 2021-03-24T20:46:00.964 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.961+0000 7fef6b705080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.965 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.961+0000 7fef6b705080 20 rados_obj.operate() r=0 bl.length=855 2021-03-24T20:46:00.965 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.961+0000 7fef6b705080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.966 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.961+0000 7fef6b705080 20 rados_obj.operate() r=-2 bl.length=0 2021-03-24T20:46:00.966 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.961+0000 7fef6b705080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.966 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.961+0000 7fef6b705080 20 rados_obj.operate() r=0 bl.length=46 2021-03-24T20:46:00.966 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.961+0000 7fef6b705080 20 rados->read ofs=0 len=0 2021-03-24T20:46:00.967 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:00.965+0000 7fef6b705080 20 rados_obj.operate() r=0 bl.length=358 2021-03-24T20:46:00.976 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"ae4b6791-a8a5-4095-9b94-989035cfb1b5","name":"default","domain_root":"default.rgw.meta:root","control_pool":"default.rgw.control","gc_pool":"default.rgw.log:gc","lc_pool":"default.rgw.log:lc","log_pool":"default.rgw.log","intent_log_pool":"default.rgw.log:intent","usage_log_pool":"default.rgw.log:usage","roles_pool":"default.rgw.meta:roles","reshard_pool":"default.rgw.log:reshard","user_keys_pool":"default.rgw.meta:users.keys","user_email_pool":"default.rgw.meta:users.email","user_swift_pool":"default.rgw.meta:users.swift","user_uid_pool":"default.rgw.meta:users.uid","otp_pool":"default.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"default.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"default.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"default.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-24T20:46:00.977 DEBUG:tasks.util.rgw: json result: {'id': 'ae4b6791-a8a5-4095-9b94-989035cfb1b5', 'name': 'default', 'domain_root': 'default.rgw.meta:root', 'control_pool': 'default.rgw.control', 'gc_pool': 'default.rgw.log:gc', 'lc_pool': 'default.rgw.log:lc', 'log_pool': 'default.rgw.log', 'intent_log_pool': 'default.rgw.log:intent', 'usage_log_pool': 'default.rgw.log:usage', 'roles_pool': 'default.rgw.meta:roles', 'reshard_pool': 'default.rgw.log:reshard', 'user_keys_pool': 'default.rgw.meta:users.keys', 'user_email_pool': 'default.rgw.meta:users.email', 'user_swift_pool': 'default.rgw.meta:users.swift', 'user_uid_pool': 'default.rgw.meta:users.uid', 'otp_pool': 'default.rgw.otp', 'system_key': {'access_key': '', 'secret_key': ''}, 'placement_pools': [{'key': 'default-placement', 'val': {'index_pool': 'default.rgw.buckets.index', 'storage_classes': {'STANDARD': {'data_pool': 'default.rgw.buckets.data', 'compression_type': 'random'}}, 'data_extra_pool': 'default.rgw.buckets.non-ec', 'index_type': 0}}], 'realm_id': ''} 2021-03-24T20:46:00.977 INFO:tasks.rgw:Starting rgw... 2021-03-24T20:46:00.978 INFO:tasks.rgw:rgw c1.client.0 config is {'port': 8000} 2021-03-24T20:46:00.978 INFO:tasks.rgw:Using beast as radosgw frontend 2021-03-24T20:46:00.978 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-03-24T20:46:00.981 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-03-24T20:46:00.984 INFO:tasks.rgw.c1.client.0:Started 2021-03-24T20:46:00.985 INFO:tasks.rgw:rgw c2.client.0 config is {'port': 8000} 2021-03-24T20:46:00.985 INFO:tasks.rgw:Using beast as radosgw frontend 2021-03-24T20:46:00.985 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-03-24T20:46:00.986 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-03-24T20:46:01.020 INFO:tasks.rgw.c2.client.0:Started 2021-03-24T20:46:01.020 INFO:tasks.rgw:rgw c2.client.1 config is {'port': 8001} 2021-03-24T20:46:01.020 INFO:tasks.rgw:Using beast as radosgw frontend 2021-03-24T20:46:01.021 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-03-24T20:46:01.021 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-03-24T20:46:01.023 INFO:tasks.rgw.c2.client.1:Started 2021-03-24T20:46:01.024 INFO:tasks.rgw:Polling c1.client.0 until it starts accepting connections on http://smithi052.front.sepia.ceph.com:8000/ 2021-03-24T20:46:01.024 DEBUG:teuthology.orchestra.run.smithi052:> curl http://smithi052.front.sepia.ceph.com:8000/ 2021-03-24T20:46:01.052 INFO:teuthology.orchestra.run.smithi052.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-24T20:46:01.052 INFO:teuthology.orchestra.run.smithi052.stderr: Dload Upload Total Spent Left Speed 2021-03-24T20:46:01.053 INFO:teuthology.orchestra.run.smithi052.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi052.front.sepia.ceph.com port 8000: Connection refused 2021-03-24T20:46:01.054 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-03-24T20:46:02.056 DEBUG:teuthology.orchestra.run.smithi052:> curl http://smithi052.front.sepia.ceph.com:8000/ 2021-03-24T20:46:02.074 INFO:teuthology.orchestra.run.smithi052.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-24T20:46:02.074 INFO:teuthology.orchestra.run.smithi052.stderr: Dload Upload Total Spent Left Speed 2021-03-24T20:46:02.079 INFO:teuthology.orchestra.run.smithi052.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 42800 2021-03-24T20:46:02.081 INFO:teuthology.orchestra.run.smithi052.stdout:anonymous 2021-03-24T20:46:02.164 INFO:tasks.rgw:Polling c2.client.0 until it starts accepting connections on http://smithi063.front.sepia.ceph.com:8000/ 2021-03-24T20:46:02.165 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8000/ 2021-03-24T20:46:02.185 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-24T20:46:02.187 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2021-03-24T20:46:02.191 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 42800 2021-03-24T20:46:02.192 INFO:teuthology.orchestra.run.smithi063.stdout:anonymous 2021-03-24T20:46:02.192 INFO:tasks.rgw:Polling c2.client.1 until it starts accepting connections on http://smithi063.front.sepia.ceph.com:8001/ 2021-03-24T20:46:02.193 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8001/ 2021-03-24T20:46:02.248 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-24T20:46:02.248 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2021-03-24T20:46:02.253 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2021-03-24T20:46:02.254 INFO:teuthology.orchestra.run.smithi063.stdout:anonymous 2021-03-24T20:46:02.255 INFO:teuthology.run_tasks:Running task rgw-multisite... 2021-03-24T20:46:02.734 INFO:tasks.rgw_multisite:creating realm.. 2021-03-24T20:46:02.735 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:02.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'realm', 'create', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:02.737 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 realm create --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T20:46:02.775 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:46:02.775 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:46:02.826 INFO:teuthology.orchestra.run.smithi052.stdout:{"id":"e90890b4-d954-49f4-b444-8f808b8250e8","name":"test-realm","current_period":"f8fe4b35-2e2b-4252-9fc7-57fba3c98b85","epoch":1} 2021-03-24T20:46:02.827 INFO:tasks.rgw_multisite:creating master zonegroup.. 2021-03-24T20:46:02.827 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:02.828 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'set', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:02.828 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup set --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T20:46:02.907 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:46:02.907 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:46:02.930 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:46:02.925+0000 7fa577b36080 -1 old format 2021-03-24T20:46:02.946 INFO:teuthology.orchestra.run.smithi052.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","sync_policy":{"groups":[]}} 2021-03-24T20:46:02.947 INFO:tasks.rgw_multisite:creating master zone.. 2021-03-24T20:46:02.947 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'HFYQ9V2T1L1XP8R6', '--secret', 'EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS', '--endpoints', 'http://smithi052.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:02.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--master', '--access-key', 'HFYQ9V2T1L1XP8R6', '--secret', 'EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS', '--endpoints', 'http://smithi052.front.sepia.ceph.com:8000', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:02.948 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --master --access-key HFYQ9V2T1L1XP8R6 --secret EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS --endpoints http://smithi052.front.sepia.ceph.com:8000 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T20:46:03.035 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:46:03.039 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:46:03.103 INFO:teuthology.orchestra.run.smithi052.stdout:{"id":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"HFYQ9V2T1L1XP8R6","secret_key":"EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8"} 2021-03-24T20:46:03.104 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool create test-zone1.rgw.buckets.data 64 64 --cluster c1 2021-03-24T20:46:04.377 INFO:teuthology.orchestra.run.smithi052.stderr:pool 'test-zone1.rgw.buckets.data' created 2021-03-24T20:46:04.395 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph osd pool application enable test-zone1.rgw.buckets.data rgw --cluster c1 2021-03-24T20:46:05.385 INFO:teuthology.orchestra.run.smithi052.stderr:enabled application 'rgw' on pool 'test-zone1.rgw.buckets.data' 2021-03-24T20:46:05.408 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:05.409 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'placement', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:05.409 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone placement --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T20:46:05.446 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:46:05.446 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:46:05.472 INFO:teuthology.orchestra.run.smithi052.stdout:{"id":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"HFYQ9V2T1L1XP8R6","secret_key":"EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-24T20:46:05.472 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:05.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:05.473 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T20:46:05.553 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:46:05.554 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:46:13.572 INFO:teuthology.orchestra.run.smithi052.stdout:{"id":"ded1abf8-1b45-4199-a116-504fb1354378","epoch":1,"predecessor_uuid":"f8fe4b35-2e2b-4252-9fc7-57fba3c98b85","sync_status":[],"period_map":{"id":"ded1abf8-1b45-4199-a116-504fb1354378","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","zones":[{"id":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","name":"test-zone1","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","val":2285303741}]},"master_zonegroup":"test-zonegroup","master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","realm_name":"test-realm","realm_epoch":2} 2021-03-24T20:46:13.573 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-03-24T20:46:13.578 INFO:teuthology.orchestra.run:waiting for 300 2021-03-24T20:46:13.615 INFO:tasks.rgw.c1.client.0.smithi052.stdout:2021-03-24T20:46:13.613+0000 7faf0567b700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground (PID: 14022) UID: 0 2021-03-24T20:46:13.616 INFO:tasks.rgw.c1.client.0.smithi052.stdout:2021-03-24T20:46:13.613+0000 7faf1cd400c0 -1 shutting down 2021-03-24T20:46:16.749 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-03-24T20:46:19.680 INFO:tasks.rgw.c1.client.0:Stopped 2021-03-24T20:46:19.680 INFO:tasks.rgw.c1.client.0:Restarting daemon 2021-03-24T20:46:19.681 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c1.client.0.stdout 2>&1 2021-03-24T20:46:19.684 INFO:tasks.rgw.c1.client.0:Started 2021-03-24T20:46:19.685 DEBUG:teuthology.orchestra.run.smithi052:> curl http://smithi052.front.sepia.ceph.com:8000 2021-03-24T20:46:19.702 INFO:teuthology.orchestra.run.smithi052.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-24T20:46:19.703 INFO:teuthology.orchestra.run.smithi052.stderr: Dload Upload Total Spent Left Speed 2021-03-24T20:46:19.706 INFO:teuthology.orchestra.run.smithi052.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi052.front.sepia.ceph.com port 8000: Connection refused 2021-03-24T20:46:19.707 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-03-24T20:46:20.709 DEBUG:teuthology.orchestra.run.smithi052:> curl http://smithi052.front.sepia.ceph.com:8000 2021-03-24T20:46:20.726 INFO:teuthology.orchestra.run.smithi052.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-24T20:46:20.726 INFO:teuthology.orchestra.run.smithi052.stderr: Dload Upload Total Spent Left Speed 2021-03-24T20:46:20.731 INFO:teuthology.orchestra.run.smithi052.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 35666 0 --:--:-- --:--:-- --:--:-- 42800 2021-03-24T20:46:20.732 INFO:teuthology.orchestra.run.smithi052.stdout:anonymous 2021-03-24T20:46:20.733 INFO:tasks.rgw_multisite:creating admin user.. 2021-03-24T20:46:20.733 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'HFYQ9V2T1L1XP8R6', '--secret', 'EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:20.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'realm-admin', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Realm Admin', '--system', '--access-key', 'HFYQ9V2T1L1XP8R6', '--secret', 'EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:20.734 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid realm-admin --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Realm Admin' --system --access-key HFYQ9V2T1L1XP8R6 --secret EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T20:46:20.828 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:46:20.828 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:46:20.984 INFO:teuthology.orchestra.run.smithi052.stdout:{"user_id":"realm-admin","display_name":"Realm Admin","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"realm-admin","access_key":"HFYQ9V2T1L1XP8R6","secret_key":"EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","system":"true","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2021-03-24T20:46:21.011 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-03-24T20:46:21.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi052.front.sepia.ceph.com:8000', '--access-key', 'HFYQ9V2T1L1XP8R6', '--secret', 'EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:21.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi052.front.sepia.ceph.com:8000', '--access-key', 'HFYQ9V2T1L1XP8R6', '--secret', 'EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:21.013 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi052.front.sepia.ceph.com:8000 --access-key HFYQ9V2T1L1XP8R6 --secret EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:46:21.056 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:46:21.056 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:46:21.169 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"e90890b4-d954-49f4-b444-8f808b8250e8","name":"test-realm","current_period":"ded1abf8-1b45-4199-a116-504fb1354378","epoch":2} 2021-03-24T20:46:21.170 INFO:tasks.rgw_multisite:creating zone.. 2021-03-24T20:46:21.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'HFYQ9V2T1L1XP8R6', '--secret', 'EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS', '--endpoints', 'http://smithi063.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:21.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--default', '--access-key', 'HFYQ9V2T1L1XP8R6', '--secret', 'EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS', '--endpoints', 'http://smithi063.front.sepia.ceph.com:8000', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:21.171 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --default --access-key HFYQ9V2T1L1XP8R6 --secret EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS --endpoints http://smithi063.front.sepia.ceph.com:8000 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:46:21.257 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:46:21.257 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:46:21.312 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:21.309+0000 7fa717d04080 0 failed reading obj info from .rgw.root:zone_info.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df: (2) No such file or directory 2021-03-24T20:46:21.312 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:21.309+0000 7fa717d04080 0 WARNING: could not read zone params for zone id=662a64c9-f3f4-4dbc-9258-dc3b70a5b4df name=test-zone1 2021-03-24T20:46:21.325 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"HFYQ9V2T1L1XP8R6","secret_key":"EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8"} 2021-03-24T20:46:21.326 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool create test-zone2.rgw.buckets.data 64 64 --cluster c2 2021-03-24T20:46:21.984 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'test-zone2.rgw.buckets.data' created 2021-03-24T20:46:22.002 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool application enable test-zone2.rgw.buckets.data rgw --cluster c2 2021-03-24T20:46:22.358 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-03-24T20:46:23.012 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rgw' on pool 'test-zone2.rgw.buckets.data' 2021-03-24T20:46:23.036 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:23.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:23.037 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:46:23.070 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:46:23.161 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:46:23.163 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"HFYQ9V2T1L1XP8R6","secret_key":"EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":""} 2021-03-24T20:46:23.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:23.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:23.165 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:46:23.244 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:46:23.244 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:46:31.174 INFO:teuthology.orchestra.run.smithi063.stderr:Sending period to new master zone 662a64c9-f3f4-4dbc-9258-dc3b70a5b4df 2021-03-24T20:46:32.259 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"ded1abf8-1b45-4199-a116-504fb1354378","epoch":2,"predecessor_uuid":"f8fe4b35-2e2b-4252-9fc7-57fba3c98b85","sync_status":[],"period_map":{"id":"ded1abf8-1b45-4199-a116-504fb1354378","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","zones":[{"id":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","name":"test-zone1","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","endpoints":["http://smithi063.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","val":2285303741},{"key":"b3573a14-e2bf-4e98-b96a-79727832ba59","val":746531257}]},"master_zonegroup":"test-zonegroup","master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","realm_name":"test-realm","realm_epoch":2} 2021-03-24T20:46:32.260 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-03-24T20:46:32.260 INFO:teuthology.orchestra.run:waiting for 300 2021-03-24T20:46:32.302 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T20:46:32.301+0000 7f09d7fff700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground (PID: 14679) UID: 0 2021-03-24T20:46:32.303 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T20:46:32.301+0000 7f09f3b030c0 -1 shutting down 2021-03-24T20:46:34.304 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-03-24T20:46:38.363 INFO:tasks.rgw.c2.client.0:Stopped 2021-03-24T20:46:38.363 INFO:tasks.rgw.c2.client.0:Restarting daemon 2021-03-24T20:46:38.364 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8000' -n client.0 --cluster c2 -k /etc/ceph/c2.client.0.keyring --log-file /var/log/ceph/rgw.c2.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock --foreground --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.0.stdout 2>&1 2021-03-24T20:46:38.367 INFO:tasks.rgw.c2.client.0:Started 2021-03-24T20:46:38.367 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8000 2021-03-24T20:46:38.383 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-24T20:46:38.384 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2021-03-24T20:46:38.387 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi063.front.sepia.ceph.com port 8000: Connection refused 2021-03-24T20:46:38.388 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-03-24T20:46:39.389 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8000 2021-03-24T20:46:39.407 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-24T20:46:39.408 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2021-03-24T20:46:39.413 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 214 0 214 0 0 42800 0 --:--:-- --:--:-- --:--:-- 42800 2021-03-24T20:46:39.414 INFO:teuthology.orchestra.run.smithi063.stdout:anonymous 2021-03-24T20:46:39.414 INFO:tasks.rgw_multisite:pulling realm configuration to c2 2021-03-24T20:46:39.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi052.front.sepia.ceph.com:8000', '--access-key', 'HFYQ9V2T1L1XP8R6', '--secret', 'EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS', '--url', 'http://smithi052.front.sepia.ceph.com:8000', '--access-key', 'HFYQ9V2T1L1XP8R6', '--secret', 'EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:39.415 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'realm', 'pull', '--rgw-realm', 'test-realm', '--url', 'http://smithi052.front.sepia.ceph.com:8000', '--access-key', 'HFYQ9V2T1L1XP8R6', '--secret', 'EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS', '--url', 'http://smithi052.front.sepia.ceph.com:8000', '--access-key', 'HFYQ9V2T1L1XP8R6', '--secret', 'EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:39.416 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 realm pull --rgw-realm test-realm --url http://smithi052.front.sepia.ceph.com:8000 --access-key HFYQ9V2T1L1XP8R6 --secret EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS --url http://smithi052.front.sepia.ceph.com:8000 --access-key HFYQ9V2T1L1XP8R6 --secret EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:46:39.508 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:46:39.508 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:46:39.553 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T20:46:39.549+0000 7f2b1f62e700 -1 rgw realm watcher: RGWRealmWatcher::handle_error oid=realms.e90890b4-d954-49f4-b444-8f808b8250e8.control err=-107 2021-03-24T20:46:39.575 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"e90890b4-d954-49f4-b444-8f808b8250e8","name":"test-realm","current_period":"ded1abf8-1b45-4199-a116-504fb1354378","epoch":2} 2021-03-24T20:46:39.576 INFO:tasks.rgw_multisite:creating zone.. 2021-03-24T20:46:39.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HFYQ9V2T1L1XP8R6', '--secret', 'EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS', '--endpoints', 'http://smithi063.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:39.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HFYQ9V2T1L1XP8R6', '--secret', 'EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS', '--endpoints', 'http://smithi063.front.sepia.ceph.com:8001', '--tier-type', 'pubsub', '--sync-from-all=0', '--sync-from', 'test-zone1', '--tier-config', 'start_with_full_sync=false,event_retention_days=7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:39.579 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key HFYQ9V2T1L1XP8R6 --secret EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS --endpoints http://smithi063.front.sepia.ceph.com:8001 --tier-type pubsub --sync-from-all=0 --sync-from test-zone1 --tier-config start_with_full_sync=false,event_retention_days=7 --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:46:39.628 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:46:39.629 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:46:39.688 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:39.685+0000 7f3eee85e080 0 failed reading obj info from .rgw.root:zone_info.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df: (2) No such file or directory 2021-03-24T20:46:39.689 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:46:39.685+0000 7f3eee85e080 0 WARNING: could not read zone params for zone id=662a64c9-f3f4-4dbc-9258-dc3b70a5b4df name=test-zone1 2021-03-24T20:46:39.701 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"91064457-7aa0-4413-b153-d6cf440dd094","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"HFYQ9V2T1L1XP8R6","secret_key":"EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8"} 2021-03-24T20:46:39.701 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool create test-zone3.rgw.buckets.data 64 64 --cluster c2 2021-03-24T20:46:40.010 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 has been restored 2021-03-24T20:46:40.102 INFO:teuthology.orchestra.run.smithi063.stderr:pool 'test-zone3.rgw.buckets.data' created 2021-03-24T20:46:40.117 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph osd pool application enable test-zone3.rgw.buckets.data rgw --cluster c2 2021-03-24T20:46:41.112 INFO:teuthology.orchestra.run.smithi063.stderr:enabled application 'rgw' on pool 'test-zone3.rgw.buckets.data' 2021-03-24T20:46:41.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:41.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'placement', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', 'modify', '--placement-id', 'default-placement', '--compression', 'random', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:41.127 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone placement --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm modify --placement-id default-placement --compression random --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:46:41.172 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:46:41.173 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:46:41.199 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"91064457-7aa0-4413-b153-d6cf440dd094","name":"test-zone3","domain_root":"test-zone3.rgw.meta:root","control_pool":"test-zone3.rgw.control","gc_pool":"test-zone3.rgw.log:gc","lc_pool":"test-zone3.rgw.log:lc","log_pool":"test-zone3.rgw.log","intent_log_pool":"test-zone3.rgw.log:intent","usage_log_pool":"test-zone3.rgw.log:usage","roles_pool":"test-zone3.rgw.meta:roles","reshard_pool":"test-zone3.rgw.log:reshard","user_keys_pool":"test-zone3.rgw.meta:users.keys","user_email_pool":"test-zone3.rgw.meta:users.email","user_swift_pool":"test-zone3.rgw.meta:users.swift","user_uid_pool":"test-zone3.rgw.meta:users.uid","otp_pool":"test-zone3.rgw.otp","system_key":{"access_key":"HFYQ9V2T1L1XP8R6","secret_key":"EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone3.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone3.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone3.rgw.buckets.non-ec","index_type":0}}],"tier_config":{"event_retention_days":7,"start_with_full_sync":false},"realm_id":""} 2021-03-24T20:46:41.199 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:41.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:41.200 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:46:41.281 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:46:41.281 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:46:49.317 INFO:teuthology.orchestra.run.smithi063.stderr:Sending period to new master zone 662a64c9-f3f4-4dbc-9258-dc3b70a5b4df 2021-03-24T20:46:50.577 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"ded1abf8-1b45-4199-a116-504fb1354378","epoch":3,"predecessor_uuid":"f8fe4b35-2e2b-4252-9fc7-57fba3c98b85","sync_status":[],"period_map":{"id":"ded1abf8-1b45-4199-a116-504fb1354378","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","zones":[{"id":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","name":"test-zone1","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"91064457-7aa0-4413-b153-d6cf440dd094","name":"test-zone3","endpoints":["http://smithi063.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","endpoints":["http://smithi063.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","val":2285303741},{"key":"91064457-7aa0-4413-b153-d6cf440dd094","val":575160010},{"key":"b3573a14-e2bf-4e98-b96a-79727832ba59","val":746531257}]},"master_zonegroup":"test-zonegroup","master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","realm_name":"test-realm","realm_epoch":2} 2021-03-24T20:46:50.578 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-03-24T20:46:50.579 INFO:teuthology.orchestra.run:waiting for 300 2021-03-24T20:46:50.623 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T20:46:50.621+0000 7fbc39a5c700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground (PID: 14682) UID: 0 2021-03-24T20:46:50.623 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T20:46:50.621+0000 7fbc509200c0 -1 shutting down 2021-03-24T20:46:51.323 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-03-24T20:46:56.681 INFO:tasks.rgw.c2.client.1:Stopped 2021-03-24T20:46:56.690 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-03-24T20:46:56.691 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-03-24T20:46:56.693 INFO:tasks.rgw.c2.client.1:Started 2021-03-24T20:46:56.694 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8001 2021-03-24T20:46:56.715 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-24T20:46:56.716 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2021-03-24T20:46:56.718 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi063.front.sepia.ceph.com port 8001: Connection refused 2021-03-24T20:46:56.719 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-03-24T20:46:57.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2021-03-24T20:46:57.721 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8001 2021-03-24T20:46:57.738 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-24T20:46:57.739 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2021-03-24T20:46:57.741 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 26800 0 --:--:-- --:--:-- --:--:-- 33500 2021-03-24T20:46:57.743 INFO:teuthology.orchestra.run.smithi063.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-00605ba541-10e6-test-zone3","HostId":"10e6-test-zone3-test-zonegroup"} 2021-03-24T20:46:57.743 INFO:tasks.rgw_multisite:rgw multisite configuration completed 2021-03-24T20:46:57.744 INFO:teuthology.run_tasks:Running task rgw-multisite-tests... 2021-03-24T20:46:57.760 INFO:tasks.rgw_multisite_tests:cloning suite branch octopus from https://github.com/ceph/ceph.git... 2021-03-24T20:46:57.768 DEBUG:teuthology.repo_utils:Setting repo remote to https://github.com/ceph/ceph.git 2021-03-24T20:46:57.789 INFO:teuthology.repo_utils:Fetching octopus from origin 2021-03-24T20:46:59.073 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/github.com_ceph_ceph_octopus to origin/octopus 2021-03-24T20:46:59.947 INFO:tasks.rgw_multisite_tests:importing tests from /home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi 2021-03-24T20:46:59.974 INFO:tasks.rgw_multisite_tests:creating test user.. 2021-03-24T20:46:59.975 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:59.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'user', 'create', '--uid', 'rgw-multisite-test-user', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--display-name', 'Multisite Test User', '--gen-access-key', '--gen-secret', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:46:59.977 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 user create --uid rgw-multisite-test-user --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --display-name 'Multisite Test User' --gen-access-key --gen-secret --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T20:47:00.024 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:47:00.025 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:47:00.156 INFO:teuthology.orchestra.run.smithi052.stdout:{"user_id":"rgw-multisite-test-user","display_name":"Multisite Test User","email":"","suspended":0,"max_buckets":1000,"subusers":[],"keys":[{"user":"rgw-multisite-test-user","access_key":"GWZZ2A5NYV9Q0BIKXMSN","secret_key":"VbrJfucqM2dxMlXyvZuDDJXuZVqfD63bPyuoo1DF"}],"swift_keys":[],"caps":[],"op_mask":"read, write, delete","default_placement":"","default_storage_class":"","placement_tags":[],"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"temp_url_keys":[],"type":"rgw","mfa_ids":[]} 2021-03-24T20:47:00.181 INFO:rgw_multi.tests:meta checkpoint 2021-03-24T20:47:00.181 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:00.182 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:00.182 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:47:00.264 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:47:00.264 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:47:00.415 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:47:00.445 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-24T20:47:00.445 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:47:00.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:00.446 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:00.447 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:47:00.487 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:47:00.487 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:47:00.680 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:47:00.681 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:47:05.682 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:05.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:05.685 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:47:05.729 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:47:05.730 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:47:05.934 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:47:05.935 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:47:10.936 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:10.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:10.938 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:47:10.979 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:47:10.979 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:47:11.176 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:47:11.177 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:47:16.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:16.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:16.179 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:47:16.219 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:47:16.220 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:47:16.409 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:47:16.410 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:47:21.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:21.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:21.414 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:47:21.459 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:47:21.459 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:47:21.652 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:47:21.653 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:47:26.654 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:26.657 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:26.658 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:47:26.698 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:47:26.699 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:47:26.904 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:47:26.905 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:47:31.906 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:31.907 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:31.908 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:47:31.955 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:47:31.955 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:47:32.158 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:47:32.158 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:47:37.159 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:37.160 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:37.160 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:47:37.198 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:47:37.199 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:47:37.410 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:47:37.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:47:42.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:42.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:42.413 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:47:42.456 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:47:42.456 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:47:42.656 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:47:42.657 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:47:47.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:47.659 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:47.659 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:47:47.701 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:47:47.702 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:47:47.910 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:47:47.911 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:47:52.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:52.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:52.915 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:47:52.955 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:47:52.955 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:47:53.150 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:47:53.151 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:47:58.152 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:58.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:47:58.153 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:47:58.194 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:47:58.195 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:47:58.378 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:47:58.379 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:48:03.381 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:03.393 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:03.393 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:48:03.438 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:48:03.438 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:48:03.638 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:48:03.661 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:48:08.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:08.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:08.671 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:48:08.719 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:48:08.719 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:48:08.931 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:48:08.932 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:48:13.933 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:13.934 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:13.934 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:48:13.974 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:48:13.975 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:48:14.179 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:48:14.180 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:48:19.181 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:19.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:19.182 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:48:19.222 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:48:19.223 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:48:19.405 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:48:19.406 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:48:24.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:24.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:24.411 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:48:24.456 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:48:24.457 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:48:24.659 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:48:24.660 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:48:29.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:29.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:29.663 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:48:29.707 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:48:29.707 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:48:29.910 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:48:29.911 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:48:34.912 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:34.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:34.913 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:48:34.954 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:48:34.955 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:48:35.138 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:48:35.139 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:48:40.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:40.141 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:40.142 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:48:40.179 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:48:40.180 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:48:40.393 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:48:40.393 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:48:45.394 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:45.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:45.395 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:48:45.437 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:48:45.437 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:48:45.651 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:48:45.652 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:48:50.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:50.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:50.654 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:48:50.694 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:48:50.695 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:48:50.889 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:48:50.890 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=0 master=1_1616618820.151371_27.1 target= 2021-03-24T20:48:55.890 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:55.891 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:55.892 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:48:55.934 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:48:55.934 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:48:56.132 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:48:56.132 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:48:56.133 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:48:56.133 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:56.134 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:56.134 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:48:56.214 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:48:56.214 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:48:56.474 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:48:56.475 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:48:56.475 INFO:rgw_multi.tests:meta checkpoint 2021-03-24T20:48:56.476 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:56.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:56.477 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:48:56.514 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:48:56.515 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:48:56.662 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:48:56.691 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"}] 2021-03-24T20:48:56.692 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:48:56.692 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:56.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:56.693 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:48:56.733 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:48:56.733 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:48:56.930 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:48:56.931 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:48:56.931 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:48:56.932 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:56.932 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:48:56.932 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:48:57.012 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:48:57.013 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:48:57.266 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:48:57.266 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:48:57.267 INFO:tasks.rgw_multisite_tests:running rgw multisite tests on '/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi' with args=['tests.py', 'tests_ps.py'] 2021-03-24T20:48:57.335 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-1 2021-03-24T20:49:00.545 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-2 2021-03-24T20:49:04.240 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:04.240 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:04.241 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:04.281 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:04.282 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:04.420 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:49:04.446 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616618940.572891_37.1","last_update":"2021-03-24T20:49:00.572891Z"},{"marker":"","last_update":"0.000000"}] 2021-03-24T20:49:04.447 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:49:04.448 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:04.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:04.450 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:04.494 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:04.495 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:04.705 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618940.572891_37.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:00.572891Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:04.705 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:49:04.706 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:49:04.706 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:04.707 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:04.707 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:04.789 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:04.789 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:05.049 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618940.525007_30.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:00.525007Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:05.050 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=2 master=1_1616618940.572891_37.1 target=1_1616618940.525007_30.1 2021-03-24T20:49:10.051 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:10.052 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:10.053 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:10.094 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:10.095 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:10.352 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618940.572891_37.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:00.572891Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:10.353 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:49:10.373 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_create ... ok 2021-03-24T20:49:10.377 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-3 2021-03-24T20:49:10.405 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-4 2021-03-24T20:49:10.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:10.452 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:10.453 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:10.493 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:10.493 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:10.640 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:49:10.666 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616618950.432415_45.1","last_update":"2021-03-24T20:49:10.432415Z"},{"marker":"","last_update":"0.000000"}] 2021-03-24T20:49:10.667 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:49:10.668 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:10.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:10.669 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:10.709 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:10.710 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:10.906 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618940.572891_37.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:00.572891Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:10.906 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1616618950.432415_45.1 target=1_1616618940.572891_37.1 2021-03-24T20:49:15.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:15.913 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:15.913 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:15.954 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:15.954 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:16.162 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618950.432415_45.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:10.432415Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:16.162 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:49:16.163 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:49:16.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:16.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:16.164 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:16.243 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:16.244 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:16.502 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618950.432415_45.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:10.432415Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:16.503 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:49:16.743 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:16.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:16.744 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:16.782 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:16.783 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:16.930 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:49:16.973 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616618956.659390_61.1","last_update":"2021-03-24T20:49:16.659390Z"},{"marker":"","last_update":"0.000000"}] 2021-03-24T20:49:16.973 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:49:16.974 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:16.975 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:16.975 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:17.016 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:17.017 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:17.201 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618950.432415_45.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:10.432415Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:17.201 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1616618956.659390_61.1 target=1_1616618950.432415_45.1 2021-03-24T20:49:22.204 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:22.220 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:22.221 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:22.264 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:22.264 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:22.463 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618956.659390_61.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:16.659390Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:22.464 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:49:22.466 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:49:22.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:22.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:22.470 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:22.546 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:22.547 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:22.805 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618956.659390_61.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:16.659390Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:22.805 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:49:22.823 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_recreate ... ok 2021-03-24T20:49:22.825 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-5 2021-03-24T20:49:22.852 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-6 2021-03-24T20:49:22.897 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:22.898 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:22.899 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:22.938 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:22.938 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:23.093 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:49:23.124 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616618962.879387_69.1","last_update":"2021-03-24T20:49:22.879387Z"},{"marker":"","last_update":"0.000000"}] 2021-03-24T20:49:23.125 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:49:23.126 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:23.127 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:23.128 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:23.169 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:23.171 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:23.369 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618956.659390_61.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:16.659390Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:23.370 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1616618962.879387_69.1 target=1_1616618956.659390_61.1 2021-03-24T20:49:28.372 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:28.379 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:28.380 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:28.421 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:28.421 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:28.620 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618962.879387_69.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:22.879387Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:28.620 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:49:28.621 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:49:28.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:28.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:28.623 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:28.705 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:28.706 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:28.967 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618962.879387_69.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:22.879387Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:29.046 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:49:29.148 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:29.149 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:29.150 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:29.187 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:29.188 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:29.347 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:49:29.375 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616618969.129804_73.1","last_update":"2021-03-24T20:49:29.129804Z"},{"marker":"","last_update":"0.000000"}] 2021-03-24T20:49:29.375 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:49:29.376 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:29.377 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:29.379 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:29.420 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:29.421 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:29.623 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618962.879387_69.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:22.879387Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:29.623 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1616618969.129804_73.1 target=1_1616618962.879387_69.1 2021-03-24T20:49:34.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:34.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:34.628 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:34.669 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:34.669 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:34.888 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618969.129804_73.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:29.129804Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:34.888 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:49:34.889 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:49:34.889 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:34.890 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:34.891 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:34.970 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:34.970 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:35.229 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618969.129804_73.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:29.129804Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:35.230 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:49:35.238 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_remove ... ok 2021-03-24T20:49:35.242 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-7 2021-03-24T20:49:35.269 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-8 2021-03-24T20:49:35.723 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:35.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:35.724 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:35.762 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:35.762 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:35.910 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:49:35.938 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616618975.295785_81.1","last_update":"2021-03-24T20:49:35.295785Z"},{"marker":"","last_update":"0.000000"}] 2021-03-24T20:49:35.939 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:49:35.940 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:35.940 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:35.941 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:35.981 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:35.982 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:36.183 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618969.129804_73.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:29.129804Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:36.184 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1616618975.295785_81.1 target=1_1616618969.129804_73.1 2021-03-24T20:49:36.376 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T20:49:36.373+0000 7f2d28ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T20:49:36.377 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T20:49:36.373+0000 7f775f7f6700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T20:49:36.377 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T20:49:36.377+0000 7f7761ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T20:49:36.378 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T20:49:36.377+0000 7f2d1a7ec700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T20:49:41.186 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:41.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:41.204 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:41.247 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:41.248 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:41.446 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618975.295785_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:35.295785Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:41.446 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:49:41.447 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:49:41.447 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:41.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:41.449 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:41.528 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:41.528 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:41.784 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618975.295785_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:35.295785Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:41.785 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:49:41.785 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:41.786 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:41.787 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-7 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:41.828 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:41.828 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:42.008 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.11.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.4.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.5.5"},{"key":8,"val":"00000000001.7.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:49:42.009 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-7 2021-03-24T20:49:42.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:42.010 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:42.010 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-7 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:42.054 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:42.055 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:42.258 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.11.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.4.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.5.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:49:42.259 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-7 2021-03-24T20:49:42.259 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-7 zones={test-zone2, test-zone1} 2021-03-24T20:49:42.279 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:49:42.290 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-24T20:49:42.290 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-24T20:49:42.291 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-24T20:49:42.291 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-24T20:49:42.292 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:49:42.339 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-24T20:49:42.436 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-24T20:49:42.437 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-24T20:49:42.437 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-24T20:49:42.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-24T20:49:42.503 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-24T20:49:42.648 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-24T20:49:42.760 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-24T20:49:42.822 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-24T20:49:42.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-24T20:49:43.033 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-24T20:49:43.092 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-24T20:49:43.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-24T20:49:43.461 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-24T20:49:43.544 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-24T20:49:43.690 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-24T20:49:43.809 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-7 zones={test-zone2, test-zone1} 2021-03-24T20:49:43.810 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:43.810 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-7', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:43.811 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-7 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:43.851 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:43.852 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:44.029 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.11.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.4.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.5.5"},{"key":8,"val":"00000000001.7.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:49:44.030 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-7 2021-03-24T20:49:44.031 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:44.031 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-7', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:44.032 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-7 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:44.078 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:44.078 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:44.335 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.11.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.4.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.5.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.7.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:49:44.336 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-7 2021-03-24T20:49:44.336 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-8', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:44.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-8', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:44.337 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-8 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:44.415 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:44.415 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:44.615 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.6.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.7.5"},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.13.5"},{"key":8,"val":"00000000001.2.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:49:44.615 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-8 2021-03-24T20:49:44.616 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-8', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:44.617 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-8', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:44.617 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-8 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:44.664 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:44.664 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:44.856 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"_myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"&","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.13.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":":","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.2.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:49:44.857 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-8 2021-03-24T20:49:44.857 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-8 zones={test-zone1, test-zone2} 2021-03-24T20:49:44.866 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:49:44.919 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-24T20:49:44.919 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-24T20:49:44.920 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-24T20:49:44.920 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-24T20:49:44.921 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:49:44.927 DEBUG:tasks.rgw_multi.zone_rados:o=& 2021-03-24T20:49:44.927 DEBUG:tasks.rgw_multi.zone_rados:o=: 2021-03-24T20:49:44.928 DEBUG:tasks.rgw_multi.zone_rados:o=_myobj 2021-03-24T20:49:44.929 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-24T20:49:44.984 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-24T20:49:45.039 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-24T20:49:45.140 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=& 2021-03-24T20:49:45.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-24T20:49:45.299 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-24T20:49:45.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=: 2021-03-24T20:49:45.517 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-24T20:49:45.575 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-24T20:49:45.686 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=_myobj 2021-03-24T20:49:45.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-24T20:49:45.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-24T20:49:45.958 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-24T20:49:46.070 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-8 zones={test-zone1, test-zone2} 2021-03-24T20:49:46.071 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_sync ... ok 2021-03-24T20:49:46.073 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-9 2021-03-24T20:49:46.100 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-10 2021-03-24T20:49:46.251 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:46.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:46.252 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:46.293 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:46.293 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:46.437 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:49:46.461 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616618986.101275_85.1","last_update":"2021-03-24T20:49:46.101275Z"},{"marker":"1_1616618986.127919_89.1","last_update":"2021-03-24T20:49:46.127919Z"}] 2021-03-24T20:49:46.461 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:49:46.462 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:46.462 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:46.463 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:46.506 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:46.506 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:46.686 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618975.295785_81.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:35.295785Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:46.687 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=2 master=1_1616618986.101275_85.1 target=1_1616618975.295785_81.1, shard=3 master=1_1616618986.127919_89.1 target= 2021-03-24T20:49:47.189 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T20:49:47.189+0000 7f2d207f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T20:49:47.190 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T20:49:47.189+0000 7f7757fe7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T20:49:47.192 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T20:49:47.189+0000 7f2d2affd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T20:49:47.192 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T20:49:47.193+0000 7f77627fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T20:49:51.689 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:51.691 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:51.691 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:51.731 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:51.732 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:51.948 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616618986.127919_89.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.127919Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:51.948 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:49:51.949 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:49:51.949 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:51.950 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:51.950 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:52.036 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:52.037 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:52.296 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616618986.127919_89.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.127919Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:49:52.297 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:49:52.297 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:52.298 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:52.298 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-9 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:52.341 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:52.342 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:52.507 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.7.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:49:52.507 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-9 2021-03-24T20:49:52.508 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:52.508 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:52.509 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-9 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:52.554 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:52.555 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:52.788 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:49:52.860 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-9 2021-03-24T20:49:52.861 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-9 zones={test-zone2, test-zone1} 2021-03-24T20:49:52.877 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:49:52.927 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-24T20:49:52.928 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:49:52.934 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-24T20:49:52.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-24T20:49:53.046 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-24T20:49:53.154 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-24T20:49:53.258 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-9 zones={test-zone2, test-zone1} 2021-03-24T20:49:53.259 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:53.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:53.260 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-9 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:53.301 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:53.302 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:53.475 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.7.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:49:53.476 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-9 2021-03-24T20:49:53.476 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:53.477 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:53.478 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-9 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:53.523 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:53.524 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:53.778 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.7.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:49:53.779 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-9 2021-03-24T20:49:53.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:53.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:53.782 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:53.876 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:53.879 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:54.047 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.5.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:49:54.048 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-10 2021-03-24T20:49:54.048 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:54.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:54.049 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:54.097 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:54.097 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:54.295 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.5.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:49:54.296 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-10 2021-03-24T20:49:54.297 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-10 zones={test-zone1, test-zone2} 2021-03-24T20:49:54.326 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:49:54.375 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-24T20:49:54.376 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:49:54.384 DEBUG:tasks.rgw_multi.zone_rados:o=myobj 2021-03-24T20:49:54.447 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-24T20:49:54.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-24T20:49:54.623 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=myobj 2021-03-24T20:49:54.767 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-10 zones={test-zone1, test-zone2} 2021-03-24T20:49:54.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:54.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:54.785 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-9 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:54.825 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:54.825 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:55.007 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.9.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:49:55.008 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-9 2021-03-24T20:49:55.009 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:55.009 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:49:55.010 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-9 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:49:55.058 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:49:55.058 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:49:55.235 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.7.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:49:55.235 WARNING:rgw_multi.tests:bucket gtvmdf-9 zone test-zone2 behind zone test-zone1: shard=1 master=00000000002.9.5 target=00000000001.7.5 2021-03-24T20:50:00.237 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:00.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:00.243 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-9 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:00.288 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:00.288 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:00.491 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.9.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:00.491 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-9 2021-03-24T20:50:00.492 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-9 zones={test-zone2, test-zone1} 2021-03-24T20:50:00.508 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:50:00.555 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:50:00.612 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-9 zones={test-zone2, test-zone1} 2021-03-24T20:50:00.613 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:00.613 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-9', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:00.614 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-9 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:00.657 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:00.657 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:00.830 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.9.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:00.831 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-9 2021-03-24T20:50:00.831 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:00.832 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-9', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:00.832 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-9 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:00.876 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:00.877 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:01.139 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.9.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:01.139 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-9 2021-03-24T20:50:01.151 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:01.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-10', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:01.152 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-10 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:01.221 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:01.409 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.8.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:01.410 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-10 2021-03-24T20:50:01.410 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:01.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:01.411 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:01.457 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:01.457 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:01.643 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.5.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:01.644 WARNING:rgw_multi.tests:bucket gtvmdf-10 zone test-zone1 behind zone test-zone2: shard=1 master=00000000002.8.5 target=00000000001.5.5 2021-03-24T20:50:06.645 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:06.647 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-10', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:06.648 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-10 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:06.693 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:06.693 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:06.886 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"myobj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.8.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:06.887 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-10 2021-03-24T20:50:06.888 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-10 zones={test-zone1, test-zone2} 2021-03-24T20:50:06.905 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:50:06.955 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:50:07.012 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-10 zones={test-zone1, test-zone2} 2021-03-24T20:50:07.013 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_object_delete ... ok 2021-03-24T20:50:07.023 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-11 2021-03-24T20:50:07.051 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-12 2021-03-24T20:50:07.125 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:07.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:07.127 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:07.171 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:07.171 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:07.321 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:50:07.350 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616618986.101275_85.1","last_update":"2021-03-24T20:49:46.101275Z"},{"marker":"1_1616619007.123420_101.1","last_update":"2021-03-24T20:50:07.123420Z"}] 2021-03-24T20:50:07.351 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:50:07.351 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:07.352 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:07.352 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:07.396 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:07.397 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:07.600 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619007.123420_101.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:07.123420Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:50:07.601 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:50:07.602 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:50:07.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:07.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:07.605 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:07.682 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:07.683 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:07.942 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619007.123420_101.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:07.123420Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:50:07.943 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:50:07.991 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:07.992 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:07.993 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:08.036 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:08.037 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:08.221 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.9.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:08.221 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-11 2021-03-24T20:50:08.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:08.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:08.223 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:08.268 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:08.268 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:08.479 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:08.480 WARNING:rgw_multi.tests:bucket gtvmdf-11 zone test-zone2 behind zone test-zone1: shard=5 master=00000000002.9.10 target= 2021-03-24T20:50:13.481 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:13.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:13.489 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:13.535 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:13.535 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:13.749 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"PYpIkUb6tM4ikDRgNgQnPHXQ5BzM1vv","ns":""},"count":1},"inc_marker":{"position":"00000000003.11.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:13.750 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-11 2021-03-24T20:50:13.750 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:13.751 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:13.752 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:13.799 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:13.800 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:13.987 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000003.11.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:14.075 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-11 2021-03-24T20:50:14.076 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:14.076 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:14.077 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:14.121 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:14.121 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:14.377 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.11.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:14.378 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-11 2021-03-24T20:50:14.431 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:14.432 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:14.433 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-12 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:14.476 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:14.476 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:14.668 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.20.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:14.669 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-12 2021-03-24T20:50:14.670 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:14.671 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:14.672 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:14.719 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:14.719 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:14.917 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:14.918 WARNING:rgw_multi.tests:bucket gtvmdf-12 zone test-zone1 behind zone test-zone2: shard=5 master=00000000002.20.10 target= 2021-03-24T20:50:19.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:19.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:19.920 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:19.964 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:19.965 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:20.167 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"IwbTe0JtZ-yZNkyBBdbLkiAmZfhV53d","ns":""},"count":1},"inc_marker":{"position":"00000000003.22.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:20.167 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-12 2021-03-24T20:50:20.918 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:20.920 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:20.920 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:20.962 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:20.962 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:21.146 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.23.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000003.11.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:21.147 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-11 2021-03-24T20:50:21.148 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:21.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:21.149 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:21.199 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:21.200 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:21.411 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"k.h0YnD2rywl34QlopN6s.obk-TF2pd","ns":""},"count":1},"inc_marker":{"position":"00000000010.23.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"PYpIkUb6tM4ikDRgNgQnPHXQ5BzM1vv","ns":""},"count":1},"inc_marker":{"position":"00000000003.11.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:21.411 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-11 2021-03-24T20:50:21.412 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:21.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:21.413 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:21.452 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:21.453 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:21.604 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.23.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000003.11.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:21.605 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-11 2021-03-24T20:50:21.605 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:21.606 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:21.606 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:21.653 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:21.654 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:21.914 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.23.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.11.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:21.915 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-11 2021-03-24T20:50:21.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:21.916 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:21.917 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:21.959 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:21.960 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:22.151 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.9.10"},{"key":2,"val":""},{"key":3,"val":"00000000010.33.5"},{"key":4,"val":""},{"key":5,"val":"00000000002.12.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:22.151 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-11 2021-03-24T20:50:22.152 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:22.153 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:22.154 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:22.203 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:22.204 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:22.382 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"k.h0YnD2rywl34QlopN6s.obk-TF2pd","ns":""},"count":1},"inc_marker":{"position":"00000000002.9.10"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"Rv56FYGtWQFs6AY7cN070psttuo3IsC","ns":""},"count":1},"inc_marker":{"position":"00000000010.33.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"PYpIkUb6tM4ikDRgNgQnPHXQ5BzM1vv","ns":""},"count":1},"inc_marker":{"position":"00000000002.12.10"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:22.383 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-11 2021-03-24T20:50:22.385 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-11 zones={test-zone2, test-zone1} 2021-03-24T20:50:22.402 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:50:22.408 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-24T20:50:22.408 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-24T20:50:22.409 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-24T20:50:22.409 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:50:22.455 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-24T20:50:22.456 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-24T20:50:22.457 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-24T20:50:22.511 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:22.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:22.588 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:22.649 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-24T20:50:22.703 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-24T20:50:22.823 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-24T20:50:22.929 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-24T20:50:22.985 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-24T20:50:23.099 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-24T20:50:23.241 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-11 zones={test-zone2, test-zone1} 2021-03-24T20:50:23.241 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:23.243 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:23.243 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:23.280 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:23.281 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:23.444 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.24.5"},{"key":2,"val":""},{"key":3,"val":"00000000002.13.10"},{"key":4,"val":""},{"key":5,"val":"00000000002.14.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:23.445 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-12 2021-03-24T20:50:23.446 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:23.447 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:23.448 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-12 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:23.502 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:23.502 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:23.694 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"cWtl0PuReKho-6n6YwoWcT--FcX-0cX","ns":""},"count":1},"inc_marker":{"position":"00000000010.24.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"RL3bJJwyc3FoCNEJ2hsUafX9E2yDVEa","ns":""},"count":1},"inc_marker":{"position":"00000000002.13.10"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"IwbTe0JtZ-yZNkyBBdbLkiAmZfhV53d","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.10"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:23.695 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-12 2021-03-24T20:50:23.696 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:23.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:23.697 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:23.752 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:23.752 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:23.932 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000010.24.5"},{"key":2,"val":""},{"key":3,"val":"00000000002.13.10"},{"key":4,"val":""},{"key":5,"val":"00000000002.14.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:23.933 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-12 2021-03-24T20:50:23.934 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:23.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:23.938 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-12 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:23.992 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:23.993 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:24.242 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.24.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.13.10"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.14.10"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:24.242 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-12 2021-03-24T20:50:24.243 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:24.244 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:24.244 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-12 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:24.322 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:24.323 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:24.508 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.13.10"},{"key":2,"val":""},{"key":3,"val":"00000000010.38.5"},{"key":4,"val":""},{"key":5,"val":"00000000003.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:24.509 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-12 2021-03-24T20:50:24.509 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:24.510 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:24.510 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:24.554 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:24.554 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:24.754 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"cWtl0PuReKho-6n6YwoWcT--FcX-0cX","ns":""},"count":1},"inc_marker":{"position":"00000000002.13.10"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"RL3bJJwyc3FoCNEJ2hsUafX9E2yDVEa","ns":""},"count":1},"inc_marker":{"position":"00000000010.38.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"IwbTe0JtZ-yZNkyBBdbLkiAmZfhV53d","ns":""},"count":1},"inc_marker":{"position":"00000000003.22.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:24.755 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-12 2021-03-24T20:50:24.756 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-12 zones={test-zone2, test-zone1} 2021-03-24T20:50:24.766 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:50:24.947 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-24T20:50:24.947 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-24T20:50:24.948 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-24T20:50:24.948 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:50:24.954 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-24T20:50:24.955 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-24T20:50:24.956 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-24T20:50:24.967 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:24.978 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:25.051 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:25.113 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-24T20:50:25.167 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-24T20:50:25.281 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-24T20:50:25.386 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-24T20:50:25.450 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-24T20:50:25.663 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-24T20:50:25.806 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-12 zones={test-zone2, test-zone1} 2021-03-24T20:50:26.121 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:26.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:26.122 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:26.163 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:26.164 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:26.348 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.27.5"},{"key":2,"val":""},{"key":3,"val":"00000000002.13.10"},{"key":4,"val":""},{"key":5,"val":"00000000003.11.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:26.349 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-11 2021-03-24T20:50:26.349 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:26.350 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:26.350 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:26.393 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:26.394 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:26.596 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"k.h0YnD2rywl34QlopN6s.obk-TF2pd","ns":""},"count":1},"inc_marker":{"position":"00000000010.23.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"Rv56FYGtWQFs6AY7cN070psttuo3IsC","ns":""},"count":1},"inc_marker":{"position":"00000000002.13.10"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"PYpIkUb6tM4ikDRgNgQnPHXQ5BzM1vv","ns":""},"count":1},"inc_marker":{"position":"00000000003.11.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:26.597 WARNING:rgw_multi.tests:bucket gtvmdf-11 zone test-zone2 behind zone test-zone1: shard=1 master=00000000011.27.5 target=00000000010.23.5 2021-03-24T20:50:31.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:31.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:31.600 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:31.642 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:31.643 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:31.848 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"k.h0YnD2rywl34QlopN6s.obk-TF2pd","ns":""},"count":1},"inc_marker":{"position":"00000000011.27.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"Rv56FYGtWQFs6AY7cN070psttuo3IsC","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"PYpIkUb6tM4ikDRgNgQnPHXQ5BzM1vv","ns":""},"count":1},"inc_marker":{"position":"00000000003.11.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:31.849 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-11 2021-03-24T20:50:31.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:31.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:31.851 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:31.888 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:31.889 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:32.055 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.27.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.16.5"},{"key":4,"val":""},{"key":5,"val":"00000000003.11.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:32.055 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-11 2021-03-24T20:50:32.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:32.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:32.057 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:32.101 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:32.101 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:32.359 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.27.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.16.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.11.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:32.359 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-11 2021-03-24T20:50:32.360 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:32.361 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-11', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:32.361 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-11 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:32.443 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:32.443 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:32.627 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.12.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.37.5"},{"key":4,"val":""},{"key":5,"val":"00000000002.12.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:32.628 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-11 2021-03-24T20:50:32.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:32.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-11', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:32.630 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-11 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:32.676 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:32.677 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:32.872 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"k.h0YnD2rywl34QlopN6s.obk-TF2pd","ns":""},"count":1},"inc_marker":{"position":"00000000003.12.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"Rv56FYGtWQFs6AY7cN070psttuo3IsC","ns":""},"count":1},"inc_marker":{"position":"00000000011.37.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"PYpIkUb6tM4ikDRgNgQnPHXQ5BzM1vv","ns":""},"count":1},"inc_marker":{"position":"00000000002.12.10"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:32.873 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-11 2021-03-24T20:50:32.874 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-11 zones={test-zone2, test-zone1} 2021-03-24T20:50:32.883 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:50:32.889 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-24T20:50:32.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-24T20:50:32.890 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-24T20:50:32.891 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:50:32.900 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-24T20:50:32.900 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-24T20:50:32.901 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-24T20:50:32.912 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:32.925 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:32.997 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:33.056 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-24T20:50:33.111 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-24T20:50:33.231 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-24T20:50:33.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-24T20:50:33.399 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-24T20:50:33.522 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-24T20:50:33.628 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-11 zones={test-zone2, test-zone1} 2021-03-24T20:50:33.629 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:33.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:33.630 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:33.670 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:33.670 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:33.855 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.28.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.16.5"},{"key":4,"val":""},{"key":5,"val":"00000000002.14.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:33.855 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-12 2021-03-24T20:50:33.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:33.856 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:33.857 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-12 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:33.906 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:33.906 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:34.108 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"cWtl0PuReKho-6n6YwoWcT--FcX-0cX","ns":""},"count":1},"inc_marker":{"position":"00000000011.28.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"RL3bJJwyc3FoCNEJ2hsUafX9E2yDVEa","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"IwbTe0JtZ-yZNkyBBdbLkiAmZfhV53d","ns":""},"count":1},"inc_marker":{"position":"00000000002.14.10"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:34.108 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-12 2021-03-24T20:50:34.109 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:34.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:34.112 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:34.157 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:34.158 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:34.343 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000011.28.5"},{"key":2,"val":""},{"key":3,"val":"00000000003.16.5"},{"key":4,"val":""},{"key":5,"val":"00000000002.14.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:34.344 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-12 2021-03-24T20:50:34.344 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:34.345 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:34.346 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-12 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:34.403 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:34.404 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:34.666 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.28.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.16.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.14.10"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:34.667 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-12 2021-03-24T20:50:34.667 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:34.668 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-12', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:34.668 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-12 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:34.747 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:34.747 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:34.931 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000003.16.5"},{"key":2,"val":""},{"key":3,"val":"00000000011.42.5"},{"key":4,"val":""},{"key":5,"val":"00000000003.22.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:34.932 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-12 2021-03-24T20:50:34.933 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:34.933 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-12', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:34.934 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-12 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:34.979 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:34.979 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:35.177 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone1","instance":"cWtl0PuReKho-6n6YwoWcT--FcX-0cX","ns":""},"count":1},"inc_marker":{"position":"00000000003.16.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj-test-zone2","instance":"RL3bJJwyc3FoCNEJ2hsUafX9E2yDVEa","ns":""},"count":1},"inc_marker":{"position":"00000000011.42.5"}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"IwbTe0JtZ-yZNkyBBdbLkiAmZfhV53d","ns":""},"count":1},"inc_marker":{"position":"00000000003.22.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:35.177 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-12 2021-03-24T20:50:35.178 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-12 zones={test-zone2, test-zone1} 2021-03-24T20:50:35.188 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:50:35.194 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-24T20:50:35.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-24T20:50:35.195 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-24T20:50:35.196 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:50:35.202 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-24T20:50:35.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone1 2021-03-24T20:50:35.203 DEBUG:tasks.rgw_multi.zone_rados:o=obj-test-zone2 2021-03-24T20:50:35.215 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:35.226 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:35.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:35.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-24T20:50:35.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-24T20:50:35.525 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone1 2021-03-24T20:50:35.629 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-24T20:50:35.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-24T20:50:35.804 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj-test-zone2 2021-03-24T20:50:35.910 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-12 zones={test-zone2, test-zone1} 2021-03-24T20:50:35.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_versioned_object_incremental_sync ... ok 2021-03-24T20:50:35.948 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:35.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:35.949 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:35.987 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:35.987 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:36.139 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:50:36.170 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616618986.101275_85.1","last_update":"2021-03-24T20:49:46.101275Z"},{"marker":"1_1616619035.950831_107.1","last_update":"2021-03-24T20:50:35.950831Z"}] 2021-03-24T20:50:36.171 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:50:36.172 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:36.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:36.175 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:36.219 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:36.219 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:36.403 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619007.123420_101.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:07.123420Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:50:36.404 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616619035.950831_107.1 target=1_1616619007.123420_101.1 2021-03-24T20:50:41.406 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:41.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:41.407 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:41.448 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:41.448 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:41.639 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619035.950831_107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:35.950831Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:50:41.639 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:50:41.640 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:50:41.641 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:41.641 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:41.642 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:41.722 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:41.722 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:41.985 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619035.950831_107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:35.950831Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:50:41.985 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:50:42.035 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:42.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:42.036 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:42.080 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:42.081 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:42.267 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.18.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:42.268 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-13 2021-03-24T20:50:42.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:42.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:42.272 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:42.318 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:42.319 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:42.507 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TYf8ljz0OYBTrjv4nPD0wEMTdLoJz80","ns":""},"count":1},"inc_marker":{"position":"00000000002.18.10"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:42.508 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-13 2021-03-24T20:50:42.508 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:42.509 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:42.509 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:42.554 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:42.554 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:42.728 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000003.20.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:42.729 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-13 2021-03-24T20:50:42.729 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:42.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:42.730 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:42.776 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:42.777 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:43.033 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.18.10"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:43.034 WARNING:rgw_multi.tests:bucket gtvmdf-13 zone test-zone3 behind zone test-zone1: shard=5 master=00000000003.20.9 target=00000000002.18.10 2021-03-24T20:50:48.035 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:48.036 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:48.037 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:48.081 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:48.081 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:48.341 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.20.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:48.342 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-13 2021-03-24T20:50:48.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:48.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:48.343 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:48.425 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:48.425 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:48.613 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.20.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:48.614 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-13 2021-03-24T20:50:48.615 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:48.615 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:48.616 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:48.662 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:48.662 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:48.859 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TYf8ljz0OYBTrjv4nPD0wEMTdLoJz80","ns":""},"count":1},"inc_marker":{"position":"00000000002.20.10"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:48.860 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-13 2021-03-24T20:50:48.860 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-13 zones={test-zone2, test-zone1} 2021-03-24T20:50:48.870 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:50:48.923 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-24T20:50:48.923 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:50:48.930 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-24T20:50:48.989 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:49.000 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:49.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:49.124 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-13 zones={test-zone2, test-zone1} 2021-03-24T20:50:50.119 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:50.119 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:50.120 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:50.164 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:50.164 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:50.340 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000003.20.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000020.42.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:50.341 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-13 2021-03-24T20:50:50.342 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:50.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:50.343 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:50.390 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:50.391 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:50.566 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TYf8ljz0OYBTrjv4nPD0wEMTdLoJz80","ns":""},"count":1},"inc_marker":{"position":"00000000003.20.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"xr77eA1mfM1WLxieVX9YAfQArZzelaz","ns":""},"count":7},"inc_marker":{"position":"00000000031.74.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:50.567 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-13 2021-03-24T20:50:50.567 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:50.568 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:50.568 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:50.610 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:50.610 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:50.788 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000003.20.9"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000031.74.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:50.789 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-13 2021-03-24T20:50:50.789 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:50.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:50.791 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:50.837 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:50.838 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:51.201 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.20.9"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000031.74.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:51.202 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-13 2021-03-24T20:50:51.203 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:51.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-13', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:51.204 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-13 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:51.245 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:51.246 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:51.422 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000002.20.10"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000031.78.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:50:51.423 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-13 2021-03-24T20:50:51.424 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:51.424 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-13', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:51.425 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-13 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:51.470 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:51.471 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:51.668 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"dummy","instance":"TYf8ljz0OYBTrjv4nPD0wEMTdLoJz80","ns":""},"count":1},"inc_marker":{"position":"00000000002.20.10"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"u3YH8xgZnPdoOMwDxwCA.xgf4LaZIGl","ns":""},"count":3},"inc_marker":{"position":"00000000031.78.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:50:51.669 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-13 2021-03-24T20:50:51.669 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-13 zones={test-zone2, test-zone1} 2021-03-24T20:50:51.679 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:50:51.689 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-24T20:50:51.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.690 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.691 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.692 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.693 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.694 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.695 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.696 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.697 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.698 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.699 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.700 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.700 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:50:51.710 DEBUG:tasks.rgw_multi.zone_rados:o=dummy 2021-03-24T20:50:51.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.711 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.712 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.713 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.714 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.715 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.716 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.717 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.718 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.719 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:50:51.737 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:51.749 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:51.815 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=dummy 2021-03-24T20:50:51.873 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:51.927 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:52.049 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:52.157 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:52.212 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:52.341 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:52.452 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:52.508 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:52.631 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:52.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:52.792 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:52.917 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:53.029 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:53.087 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:53.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:53.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:53.373 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:53.489 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:53.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:53.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:53.764 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:53.880 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:53.941 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:54.062 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:54.165 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:54.225 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:54.342 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:54.454 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:54.507 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:54.632 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:54.738 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:54.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:54.918 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:55.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:55.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:55.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:55.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:55.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:55.492 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:55.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:55.650 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:55.768 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:55.874 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:55.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:56.050 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:56.153 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:56.207 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:56.336 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:56.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:56.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:56.620 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:56.951 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:57.013 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:57.129 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:57.233 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:57.292 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:57.413 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:57.529 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:57.585 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:57.704 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:50:57.809 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-13 zones={test-zone2, test-zone1} 2021-03-24T20:50:57.809 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:57.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:57.811 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:57.853 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:57.854 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:58.031 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616619008.605164_38.1","last_update":"2021-03-24T20:50:08.605164Z"},{"marker":"1_1616619030.611627_106.2","last_update":"2021-03-24T20:50:30.611627Z"},{"marker":"1_1616619052.619482_64.1","last_update":"2021-03-24T20:50:52.619482Z"},{"marker":"1_1616619052.619482_77.1","last_update":"2021-03-24T20:50:52.619482Z"}] 2021-03-24T20:50:58.031 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-24T20:50:58.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:58.032 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:58.033 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:58.077 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:58.077 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:58.255 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619008.605164_38.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:08.605164Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619052.619482_64.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:52.619482Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619052.619482_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:52.619482Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T20:50:58.256 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-24T20:50:58.256 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:58.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:58.257 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:58.296 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:58.296 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:58.476 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616619008.605164_38.1","last_update":"2021-03-24T20:50:08.605164Z"},{"marker":"1_1616619030.611627_106.2","last_update":"2021-03-24T20:50:30.611627Z"},{"marker":"1_1616619052.619482_64.1","last_update":"2021-03-24T20:50:52.619482Z"},{"marker":"1_1616619052.619482_77.1","last_update":"2021-03-24T20:50:52.619482Z"}] 2021-03-24T20:50:58.477 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-24T20:50:58.477 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:58.478 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:58.478 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:58.522 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:58.522 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:58.785 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14224098923836610154},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619008.605164_38.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:08.605164Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619052.619482_64.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:52.619482Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619052.619482_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:52.619482Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T20:50:58.786 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-24T20:50:58.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:58.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:58.789 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:58.864 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:58.865 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:59.051 INFO:teuthology.orchestra.run.smithi063.stdout:[{"marker":"1_1616619009.557988_33.1","last_update":"2021-03-24T20:50:09.557988Z"},{"marker":"1_1616619031.564526_105.2","last_update":"2021-03-24T20:50:31.564526Z"},{"marker":"1_1616619053.571247_91.1","last_update":"2021-03-24T20:50:53.571247Z"},{"marker":"1_1616619053.571247_88.1","last_update":"2021-03-24T20:50:53.571247Z"}] 2021-03-24T20:50:59.052 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-24T20:50:59.052 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:59.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:59.053 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:59.095 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:59.095 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:59.290 INFO:teuthology.orchestra.run.smithi052.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14589021674366829938},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619009.557988_33.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:09.557988Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619031.564526_105.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:31.564526Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619053.571247_91.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:53.571247Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619053.571247_88.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:53.571247Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T20:50:59.311 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-24T20:50:59.348 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_concurrent_versioned_object_incremental_sync ... ok 2021-03-24T20:50:59.405 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:59.406 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:59.406 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:59.450 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:59.451 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:59.604 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:50:59.636 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616618986.101275_85.1","last_update":"2021-03-24T20:49:46.101275Z"},{"marker":"1_1616619059.407165_111.1","last_update":"2021-03-24T20:50:59.407165Z"}] 2021-03-24T20:50:59.637 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:50:59.638 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:59.638 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:50:59.639 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:50:59.690 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:50:59.690 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:50:59.877 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619035.950831_107.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:35.950831Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:50:59.877 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616619059.407165_111.1 target=1_1616619035.950831_107.1 2021-03-24T20:51:04.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:04.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:04.882 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:04.926 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:04.927 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:05.099 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619059.407165_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:59.407165Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:51:05.100 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:51:05.100 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:51:05.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:05.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:05.102 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:05.180 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:05.180 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:05.434 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619059.407165_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:59.407165Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:51:05.435 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:51:05.487 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:05.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:05.488 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:05.529 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:05.529 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:05.709 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.15.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:51:05.710 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-14 2021-03-24T20:51:05.710 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:05.711 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:05.712 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:05.756 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:05.757 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:05.957 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:05.957 WARNING:rgw_multi.tests:bucket gtvmdf-14 zone test-zone2 behind zone test-zone1: shard=8 master=00000000001.15.5 target= 2021-03-24T20:51:10.959 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:10.962 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:10.963 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:11.008 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:11.009 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:11.193 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:11.194 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-14 2021-03-24T20:51:11.194 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:11.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:11.196 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:11.237 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:11.238 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:11.428 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.15.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:51:11.429 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-14 2021-03-24T20:51:11.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:11.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:11.432 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:11.478 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:11.478 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:11.735 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.15.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:11.735 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-14 2021-03-24T20:51:11.736 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:11.737 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:11.738 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:11.820 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:11.820 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:12.002 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.14.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:51:12.002 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-14 2021-03-24T20:51:12.003 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:12.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:12.004 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:12.051 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:12.051 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:12.243 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.14.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:12.244 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-14 2021-03-24T20:51:12.244 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-14 zones={test-zone2, test-zone1} 2021-03-24T20:51:12.262 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:51:12.311 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:12.311 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:51:12.317 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:12.372 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:12.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:12.445 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:12.508 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-14 zones={test-zone2, test-zone1} 2021-03-24T20:51:12.520 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:12.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:12.522 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:12.562 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:12.562 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:12.719 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:51:12.746 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616618986.101275_85.1","last_update":"2021-03-24T20:49:46.101275Z"},{"marker":"1_1616619072.523480_113.1","last_update":"2021-03-24T20:51:12.523480Z"}] 2021-03-24T20:51:12.746 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:51:12.747 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:12.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:12.748 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:12.787 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:12.788 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:12.964 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619072.523480_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:51:12.523480Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:51:12.964 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:51:12.965 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:51:12.965 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:12.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:12.966 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:13.047 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:13.048 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:13.197 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619059.407165_111.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:59.407165Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:51:13.198 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1616619072.523480_113.1 target=1_1616619059.407165_111.1 2021-03-24T20:51:18.199 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:18.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:18.201 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:18.247 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:18.247 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:18.502 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619072.523480_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:51:12.523480Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:51:18.502 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:51:18.551 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:18.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:18.552 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:18.593 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:18.594 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:18.775 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.18.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:51:18.776 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-14 2021-03-24T20:51:18.777 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:18.777 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:18.778 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:18.822 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:18.823 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:19.019 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.15.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:19.020 WARNING:rgw_multi.tests:bucket gtvmdf-14 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.18.12 target=00000000001.15.5 2021-03-24T20:51:24.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:24.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:24.023 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:24.069 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:24.070 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:24.253 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:24.254 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-14 2021-03-24T20:51:24.254 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:24.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:24.256 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:24.299 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:24.300 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:24.474 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.18.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:51:24.475 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-14 2021-03-24T20:51:24.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:24.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:24.476 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:24.522 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:24.523 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:24.778 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.18.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:24.779 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-14 2021-03-24T20:51:24.780 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:24.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:24.781 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:24.864 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:24.864 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:25.066 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.18.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:51:25.067 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-14 2021-03-24T20:51:25.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:25.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:25.069 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:25.116 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:25.116 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:25.303 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:25.304 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-14 2021-03-24T20:51:25.304 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-14 zones={test-zone2, test-zone1} 2021-03-24T20:51:25.314 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:51:25.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:25.363 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:25.364 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:51:25.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:25.370 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:25.423 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:25.433 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:25.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:25.555 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:25.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:25.634 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:25.693 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-14 zones={test-zone2, test-zone1} 2021-03-24T20:51:25.705 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:25.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:25.706 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:25.745 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:25.745 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:25.890 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:51:25.920 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616618986.101275_85.1","last_update":"2021-03-24T20:49:46.101275Z"},{"marker":"1_1616619085.706880_115.1","last_update":"2021-03-24T20:51:25.706880Z"}] 2021-03-24T20:51:25.920 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:51:25.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:25.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:25.922 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:25.962 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:25.963 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:26.161 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619072.523480_113.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:51:12.523480Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:51:26.162 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616619085.706880_115.1 target=1_1616619072.523480_113.1 2021-03-24T20:51:31.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:31.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:31.165 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:31.204 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:31.204 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:31.399 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619085.706880_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:51:25.706880Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:51:31.400 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:51:31.400 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:51:31.401 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:31.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:31.402 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:31.481 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:31.482 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:31.741 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619085.706880_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:51:25.706880Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:51:31.742 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:51:31.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:31.795 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:31.796 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:31.838 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:31.839 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:32.020 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.22.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:51:32.021 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-14 2021-03-24T20:51:32.021 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:32.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:32.022 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:32.066 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:32.067 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:32.256 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000003.18.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:32.257 WARNING:rgw_multi.tests:bucket gtvmdf-14 zone test-zone2 behind zone test-zone1: shard=8 master=00000000005.22.17 target=00000000003.18.12 2021-03-24T20:51:37.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:37.259 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:37.260 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:37.306 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:37.307 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:37.520 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.22.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:37.521 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-14 2021-03-24T20:51:37.521 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:37.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:37.523 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:37.569 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:37.569 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:37.742 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.22.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:51:37.742 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-14 2021-03-24T20:51:37.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:37.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:37.744 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:37.790 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:37.791 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:38.051 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.22.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:38.052 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-14 2021-03-24T20:51:38.052 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:38.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-14', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:38.053 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-14 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:38.095 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:38.095 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:38.285 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000005.23.17"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:51:38.285 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-14 2021-03-24T20:51:38.286 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:38.286 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-14', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:38.287 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-14 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:38.332 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:38.333 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:38.513 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000005.23.17"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:38.514 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-14 2021-03-24T20:51:38.515 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-14 zones={test-zone2, test-zone1} 2021-03-24T20:51:38.524 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:51:38.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:38.571 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:38.572 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:51:38.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:38.578 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:38.636 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:38.647 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:38.714 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:38.778 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:38.791 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:38.859 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:38.917 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-14 zones={test-zone2, test-zone1} 2021-03-24T20:51:38.918 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_version_suspended_incremental_sync ... ok 2021-03-24T20:51:38.919 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-15 2021-03-24T20:51:38.945 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-16 2021-03-24T20:51:39.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:39.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:39.016 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:39.055 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:39.056 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:39.207 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:51:39.238 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616618986.101275_85.1","last_update":"2021-03-24T20:49:46.101275Z"},{"marker":"1_1616619099.012600_127.1","last_update":"2021-03-24T20:51:39.012600Z"}] 2021-03-24T20:51:39.239 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:51:39.240 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:39.241 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:39.242 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:39.282 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:39.283 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:39.480 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619085.706880_115.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:51:25.706880Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:51:39.481 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616619099.012600_127.1 target=1_1616619085.706880_115.1 2021-03-24T20:51:44.482 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:44.483 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:44.484 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:44.526 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:44.526 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:44.733 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619099.012600_127.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:51:39.012600Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:51:44.734 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:51:44.735 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:51:44.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:44.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:44.736 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:44.816 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:44.816 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:45.174 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619099.012600_127.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:51:39.012600Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:51:45.175 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:51:45.306 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:45.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:45.307 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:45.349 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:45.439 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:45.524 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.22.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:51:45.525 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-15 2021-03-24T20:51:45.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:45.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:45.526 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:45.574 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:45.574 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:45.785 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:45.786 WARNING:rgw_multi.tests:bucket gtvmdf-15 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.22.11 target= 2021-03-24T20:51:50.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:50.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:50.794 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:50.843 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:50.844 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:51.045 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"k32SDYmqoVh94ms52KeMgKhVNzmrx43","ns":""},"count":2},"inc_marker":{"position":"00000000003.22.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:51.046 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-15 2021-03-24T20:51:51.047 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:51.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:51.048 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:51.094 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:51.095 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:51.271 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.22.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:51:51.271 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-15 2021-03-24T20:51:51.272 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:51.272 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:51.273 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:51.317 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:51.318 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:51.576 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.22.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:51.576 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-15 2021-03-24T20:51:51.577 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:51.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-15', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:51.578 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-15 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:51.662 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:51.663 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:51.854 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.43.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:51:51.854 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-15 2021-03-24T20:51:51.855 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:51.855 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-15', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:51.856 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-15 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:51.900 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:51.900 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:52.095 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"k32SDYmqoVh94ms52KeMgKhVNzmrx43","ns":""},"count":2},"inc_marker":{"position":"00000000002.43.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:52.096 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-15 2021-03-24T20:51:52.097 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-15 zones={test-zone2, test-zone1} 2021-03-24T20:51:52.114 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:51:52.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:52.168 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:52.169 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:51:52.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:52.176 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:52.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:52.244 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:52.255 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:52.318 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-15 zones={test-zone2, test-zone1} 2021-03-24T20:51:52.318 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:52.319 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:52.319 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:52.361 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:52.361 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:52.529 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.12.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:51:52.530 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-16 2021-03-24T20:51:52.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:52.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:52.531 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:52.576 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:52.577 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:52.770 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"kkJqf4tshwlz.zeX7SdYak9LuRbEIaz","ns":""},"count":1},"inc_marker":{"position":"00000000002.12.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:52.771 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-16 2021-03-24T20:51:52.772 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:52.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:52.773 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:52.815 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:52.816 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:52.999 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000002.12.5"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:51:52.999 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-16 2021-03-24T20:51:53.000 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:53.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:53.001 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:53.046 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:53.047 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:53.301 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.12.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:53.302 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-16 2021-03-24T20:51:53.302 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:53.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-16', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:53.303 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-16 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:53.346 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:53.346 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:53.558 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.23.11"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:51:53.559 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-16 2021-03-24T20:51:53.559 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:53.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-16', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:53.560 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-16 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:53.604 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:53.604 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:53.805 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"kkJqf4tshwlz.zeX7SdYak9LuRbEIaz","ns":""},"count":1},"inc_marker":{"position":"00000000003.23.11"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:51:53.806 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-16 2021-03-24T20:51:53.807 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-16 zones={test-zone2, test-zone1} 2021-03-24T20:51:53.816 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:51:53.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:53.863 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:53.864 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:51:53.871 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:53.872 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:51:53.924 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:53.932 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:53.942 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:51:54.010 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-16 zones={test-zone2, test-zone1} 2021-03-24T20:51:54.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_delete_marker_full_sync ... ok 2021-03-24T20:51:54.013 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-17 2021-03-24T20:51:54.040 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-18 2021-03-24T20:51:54.138 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:54.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:54.139 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:54.178 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:54.179 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:54.331 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:51:54.361 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616618986.101275_85.1","last_update":"2021-03-24T20:49:46.101275Z"},{"marker":"1_1616619114.134086_143.1","last_update":"2021-03-24T20:51:54.134086Z"}] 2021-03-24T20:51:54.362 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:51:54.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:54.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:54.364 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:54.407 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:54.407 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:54.607 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619099.012600_127.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:51:39.012600Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:51:54.608 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616619114.134086_143.1 target=1_1616619099.012600_127.1 2021-03-24T20:51:59.609 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:59.610 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:59.610 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:59.652 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:59.653 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:51:59.859 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619114.134086_143.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:51:54.134086Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:51:59.859 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:51:59.860 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:51:59.860 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:59.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:51:59.862 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:51:59.940 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:51:59.940 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:00.202 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619114.134086_143.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:51:54.134086Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:52:00.203 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:52:00.350 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:00.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:00.352 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:00.392 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:00.393 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:00.563 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.10.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:52:00.563 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-17 2021-03-24T20:52:00.564 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:00.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:00.565 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:00.610 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:00.610 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:00.827 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:52:00.828 WARNING:rgw_multi.tests:bucket gtvmdf-17 zone test-zone2 behind zone test-zone1: shard=8 master=00000000003.10.6 target= 2021-03-24T20:52:05.829 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:05.830 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:05.831 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:05.876 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:05.876 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:06.100 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.10.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:52:06.101 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-17 2021-03-24T20:52:06.102 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:06.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:06.103 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:06.146 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:06.146 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:06.321 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.10.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:52:06.321 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-17 2021-03-24T20:52:06.322 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:06.323 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:06.323 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:06.370 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:06.370 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:06.627 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.10.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:52:06.627 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-17 2021-03-24T20:52:06.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:06.628 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-17', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:06.629 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-17 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:06.668 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:06.668 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:06.876 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.6.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:52:06.876 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-17 2021-03-24T20:52:06.877 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:06.877 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-17', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:06.878 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-17 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:06.924 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:06.925 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:07.106 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.6.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:52:07.106 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-17 2021-03-24T20:52:07.107 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-17 zones={test-zone2, test-zone1} 2021-03-24T20:52:07.115 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:52:07.167 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:52:07.167 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:52:07.173 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:52:07.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:52:07.235 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-17 zones={test-zone2, test-zone1} 2021-03-24T20:52:07.236 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:07.236 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:07.237 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:07.276 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:07.276 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:07.456 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.8.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:52:07.457 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-18 2021-03-24T20:52:07.457 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:07.458 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:07.458 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:07.502 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:07.503 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:07.712 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000001.8.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:52:07.713 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-18 2021-03-24T20:52:07.714 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:07.715 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:07.715 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:07.758 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:07.759 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:07.940 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.8.12"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:52:07.941 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-18 2021-03-24T20:52:07.942 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:07.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:07.943 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:07.986 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:07.987 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:08.246 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.8.12"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:52:08.246 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-18 2021-03-24T20:52:08.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:08.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-18', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:08.248 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-18 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:08.330 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:08.331 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:08.534 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000003.22.6"},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T20:52:08.535 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-18 2021-03-24T20:52:08.536 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:08.536 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-18', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:08.537 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-18 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:08.584 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:08.584 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:08.782 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj","instance":"null","ns":""},"count":1},"inc_marker":{"position":"00000000003.22.6"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T20:52:08.782 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-18 2021-03-24T20:52:08.783 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-18 zones={test-zone2, test-zone1} 2021-03-24T20:52:08.792 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:52:08.843 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:52:08.844 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:52:08.849 DEBUG:tasks.rgw_multi.zone_rados:o=obj 2021-03-24T20:52:08.904 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj 2021-03-24T20:52:08.910 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-18 zones={test-zone2, test-zone1} 2021-03-24T20:52:08.911 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_suspended_delete_marker_full_sync ... ok 2021-03-24T20:52:08.912 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-19 2021-03-24T20:52:08.936 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-20 2021-03-24T20:52:09.106 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_versioning ... ok 2021-03-24T20:52:09.108 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-21 2021-03-24T20:52:09.134 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-22 2021-03-24T20:52:09.299 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_acl ... ok 2021-03-24T20:52:09.300 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-23 2021-03-24T20:52:09.325 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-24 2021-03-24T20:52:10.491 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_cors ... ok 2021-03-24T20:52:10.491 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-25 2021-03-24T20:52:10.517 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-26 2021-03-24T20:52:10.562 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:10.565 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:10.565 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:10.603 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:10.603 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:10.755 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:52:10.786 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616618986.101275_85.1","last_update":"2021-03-24T20:49:46.101275Z"},{"marker":"1_1616619130.543652_187.1","last_update":"2021-03-24T20:52:10.543652Z"}] 2021-03-24T20:52:10.787 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:52:10.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:10.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:10.788 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:10.831 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:10.832 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:11.037 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619129.383378_177.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:09.383378Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:52:11.038 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616619130.543652_187.1 target=1_1616619129.383378_177.1 2021-03-24T20:52:16.039 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:16.040 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:16.041 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:16.083 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:16.173 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:16.264 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619130.543652_187.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:10.543652Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:52:16.265 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:52:16.266 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:52:16.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:16.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:16.269 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:16.346 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:16.347 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:16.609 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619130.543652_187.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:10.543652Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:52:16.609 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:52:16.896 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_delete_notempty ... ok 2021-03-24T20:52:16.897 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-27 2021-03-24T20:52:16.920 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-28 2021-03-24T20:52:16.965 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:16.966 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:16.966 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:17.007 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:17.008 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:17.156 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=ded1abf8-1b45-4199-a116-504fb1354378 2021-03-24T20:52:17.179 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616618820.151371_27.1","last_update":"2021-03-24T20:47:00.151371Z"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616618986.101275_85.1","last_update":"2021-03-24T20:49:46.101275Z"},{"marker":"1_1616619136.946453_195.1","last_update":"2021-03-24T20:52:16.946453Z"}] 2021-03-24T20:52:17.179 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:52:17.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:17.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:17.181 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:17.224 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:17.225 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:17.405 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619136.946453_195.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:16.946453Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:52:17.406 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:52:17.406 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:52:17.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:17.407 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:52:17.408 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:52:17.490 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:17.490 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:17.749 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"ded1abf8-1b45-4199-a116-504fb1354378","realm_epoch":2},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619136.946453_195.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:16.946453Z","realm_epoch":2}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:52:17.750 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:52:17.750 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-03-24T20:52:17.756 INFO:teuthology.orchestra.run:waiting for 300 2021-03-24T20:52:17.794 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T20:52:17.794+0000 7f7785839700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 17371) UID: 0 2021-03-24T20:52:17.794 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T20:52:17.794+0000 7f779cefe0c0 -1 shutting down 2021-03-24T20:52:22.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 is failed for ~0s 2021-03-24T20:52:23.858 INFO:tasks.rgw.c2.client.1:Stopped 2021-03-24T20:52:23.859 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:52:23.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:52:23.860 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:52:23.901 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:23.902 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:23.929 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:52:23.930+0000 7fd2c9756080 0 NOTICE: overriding master zone: 662a64c9-f3f4-4dbc-9258-dc3b70a5b4df 2021-03-24T20:52:23.930 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:52:23.930+0000 7fd2c9756080 0 failed reading obj info from .rgw.root:zone_info.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df: (2) No such file or directory 2021-03-24T20:52:23.931 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:52:23.930+0000 7fd2c9756080 0 WARNING: could not read zone params for zone id=662a64c9-f3f4-4dbc-9258-dc3b70a5b4df name=test-zone1 2021-03-24T20:52:23.947 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"HFYQ9V2T1L1XP8R6","secret_key":"EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8"} 2021-03-24T20:52:23.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:52:23.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:52:23.949 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:52:24.029 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:52:24.029 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:52:24.271 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"9c8fef3d-af5e-4308-a22f-c09517bb5569","epoch":1,"predecessor_uuid":"ded1abf8-1b45-4199-a116-504fb1354378","sync_status":["1_1616618820.151371_27.1","","1_1616618986.101275_85.1","1_1616619136.946453_195.1"],"period_map":{"id":"9c8fef3d-af5e-4308-a22f-c09517bb5569","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"b3573a14-e2bf-4e98-b96a-79727832ba59","zones":[{"id":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","name":"test-zone1","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"91064457-7aa0-4413-b153-d6cf440dd094","name":"test-zone3","endpoints":["http://smithi063.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","endpoints":["http://smithi063.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","val":2285303741},{"key":"91064457-7aa0-4413-b153-d6cf440dd094","val":575160010},{"key":"b3573a14-e2bf-4e98-b96a-79727832ba59","val":746531257}]},"master_zonegroup":"test-zonegroup","master_zone":"b3573a14-e2bf-4e98-b96a-79727832ba59","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","realm_name":"test-realm","realm_epoch":3} 2021-03-24T20:52:24.272 INFO:rgw_multi.tests:Set master zone=test-zone2, waiting 60s for reconfiguration.. 2021-03-24T20:52:28.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.1 has been restored 2021-03-24T20:53:24.274 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-29 2021-03-24T20:53:24.341 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-30 2021-03-24T20:53:24.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:53:24.372 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:53:24.373 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:53:24.411 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:53:24.412 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:53:24.567 INFO:teuthology.orchestra.run.smithi063.stderr:No --period given, using current period=9c8fef3d-af5e-4308-a22f-c09517bb5569 2021-03-24T20:53:24.597 INFO:teuthology.orchestra.run.smithi063.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616619204.367468_69.1","last_update":"2021-03-24T20:53:24.367468Z"}] 2021-03-24T20:53:24.598 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone1 2021-03-24T20:53:24.598 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:53:24.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:53:24.599 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:53:24.641 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:53:24.642 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:53:24.823 INFO:teuthology.orchestra.run.smithi052.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9c8fef3d-af5e-4308-a22f-c09517bb5569","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":55,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}}]},"full_sync":{"total":55,"complete":55}} 2021-03-24T20:53:24.823 WARNING:rgw_multi.tests:zone test-zone1 behind master: shard=3 master=1_1616619204.367468_69.1 target= 2021-03-24T20:53:29.825 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:53:29.826 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:53:29.827 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 metadata sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:53:29.869 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:53:29.869 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:53:30.051 INFO:teuthology.orchestra.run.smithi052.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"9c8fef3d-af5e-4308-a22f-c09517bb5569","realm_epoch":3},"markers":[{"key":0,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":55,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":2,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":3}},{"key":3,"val":{"state":1,"marker":"1_1616619204.367468_69.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:53:24.367468Z","realm_epoch":3}}]},"full_sync":{"total":55,"complete":55}} 2021-03-24T20:53:30.052 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone1 2021-03-24T20:53:30.053 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:53:30.061 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'modify', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--master', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:53:30.062 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone modify --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --master --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T20:53:30.135 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:53:30.135 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:53:30.160 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:53:30.160+0000 7fa0abaee080 0 NOTICE: overriding master zone: b3573a14-e2bf-4e98-b96a-79727832ba59 2021-03-24T20:53:30.161 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:53:30.160+0000 7fa0abaee080 0 failed reading obj info from .rgw.root:zone_info.91064457-7aa0-4413-b153-d6cf440dd094: (2) No such file or directory 2021-03-24T20:53:30.162 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:53:30.160+0000 7fa0abaee080 0 WARNING: could not read zone params for zone id=91064457-7aa0-4413-b153-d6cf440dd094 name=test-zone3 2021-03-24T20:53:30.162 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:53:30.160+0000 7fa0abaee080 0 failed reading obj info from .rgw.root:zone_info.b3573a14-e2bf-4e98-b96a-79727832ba59: (2) No such file or directory 2021-03-24T20:53:30.162 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T20:53:30.160+0000 7fa0abaee080 0 WARNING: could not read zone params for zone id=b3573a14-e2bf-4e98-b96a-79727832ba59 name=test-zone2 2021-03-24T20:53:30.173 INFO:teuthology.orchestra.run.smithi052.stdout:{"id":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","name":"test-zone1","domain_root":"test-zone1.rgw.meta:root","control_pool":"test-zone1.rgw.control","gc_pool":"test-zone1.rgw.log:gc","lc_pool":"test-zone1.rgw.log:lc","log_pool":"test-zone1.rgw.log","intent_log_pool":"test-zone1.rgw.log:intent","usage_log_pool":"test-zone1.rgw.log:usage","roles_pool":"test-zone1.rgw.meta:roles","reshard_pool":"test-zone1.rgw.log:reshard","user_keys_pool":"test-zone1.rgw.meta:users.keys","user_email_pool":"test-zone1.rgw.meta:users.email","user_swift_pool":"test-zone1.rgw.meta:users.swift","user_uid_pool":"test-zone1.rgw.meta:users.uid","otp_pool":"test-zone1.rgw.otp","system_key":{"access_key":"HFYQ9V2T1L1XP8R6","secret_key":"EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone1.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone1.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone1.rgw.buckets.non-ec","index_type":0}}],"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8"} 2021-03-24T20:53:30.174 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:53:30.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:53:30.175 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T20:53:30.253 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:53:30.254 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:53:30.494 INFO:teuthology.orchestra.run.smithi052.stdout:{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","epoch":1,"predecessor_uuid":"9c8fef3d-af5e-4308-a22f-c09517bb5569","sync_status":["","","","1_1616619204.367468_69.1"],"period_map":{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","zones":[{"id":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","name":"test-zone1","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"91064457-7aa0-4413-b153-d6cf440dd094","name":"test-zone3","endpoints":["http://smithi063.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","endpoints":["http://smithi063.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","val":2285303741},{"key":"91064457-7aa0-4413-b153-d6cf440dd094","val":575160010},{"key":"b3573a14-e2bf-4e98-b96a-79727832ba59","val":746531257}]},"master_zonegroup":"test-zonegroup","master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","realm_name":"test-realm","realm_epoch":4} 2021-03-24T20:53:30.495 INFO:rgw_multi.tests:Set master zone=test-zone1, waiting 60s for reconfiguration.. 2021-03-24T20:54:30.496 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-31 2021-03-24T20:54:30.531 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-32 2021-03-24T20:54:30.585 INFO:tasks.rgw.c2.client.1:Restarting daemon 2021-03-24T20:54:30.586 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term radosgw --rgw-frontends 'beast port=8001' -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm | sudo tee /var/log/ceph/rgw.c2.client.1.stdout 2>&1 2021-03-24T20:54:30.589 INFO:tasks.rgw.c2.client.1:Started 2021-03-24T20:54:30.589 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8001 2021-03-24T20:54:30.603 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-24T20:54:30.603 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2021-03-24T20:54:30.606 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0curl: (7) Failed to connect to smithi063.front.sepia.ceph.com port 8001: Connection refused 2021-03-24T20:54:30.607 DEBUG:teuthology.orchestra.run:got remote process result: 7 2021-03-24T20:54:31.609 DEBUG:teuthology.orchestra.run.smithi063:> curl http://smithi063.front.sepia.ceph.com:8001 2021-03-24T20:54:31.625 INFO:teuthology.orchestra.run.smithi063.stderr: % Total % Received % Xferd Average Speed Time Time Time Current 2021-03-24T20:54:31.625 INFO:teuthology.orchestra.run.smithi063.stderr: Dload Upload Total Spent Left Speed 2021-03-24T20:54:31.630 INFO:teuthology.orchestra.run.smithi063.stderr: 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 134 100 134 0 0 26800 0 --:--:-- --:--:-- --:--:-- 26800 2021-03-24T20:54:31.631 INFO:teuthology.orchestra.run.smithi063.stdout:{"Code":"MethodNotAllowed","RequestId":"tx000000000000000000001-00605ba707-1432-test-zone3","HostId":"1432-test-zone3-test-zonegroup"} 2021-03-24T20:54:31.632 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:31.632 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:31.633 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:31.671 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:31.671 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:31.824 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T20:54:31.856 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616619270.564805_223.1","last_update":"2021-03-24T20:54:30.564805Z"}] 2021-03-24T20:54:31.856 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:54:31.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:31.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:31.858 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:31.899 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:31.900 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:32.088 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619270.564805_223.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:30.564805Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:54:32.088 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:54:32.089 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:54:32.090 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:32.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:32.091 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:32.170 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:32.170 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:32.429 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619270.564805_223.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:30.564805Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:54:32.430 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:54:32.431 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-27 zones={test-zone2, test-zone1} 2021-03-24T20:54:32.446 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:54:32.495 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:54:32.552 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-27 zones={test-zone2, test-zone1} 2021-03-24T20:54:32.553 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-27 zones={test-zone1, test-zone2} 2021-03-24T20:54:32.561 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:54:32.607 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:54:32.663 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-27 zones={test-zone1, test-zone2} 2021-03-24T20:54:32.664 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-28 zones={test-zone2, test-zone1} 2021-03-24T20:54:32.675 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:54:32.727 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:54:32.788 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-28 zones={test-zone2, test-zone1} 2021-03-24T20:54:32.789 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-28 zones={test-zone1, test-zone2} 2021-03-24T20:54:32.798 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:54:32.847 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:54:32.905 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-28 zones={test-zone1, test-zone2} 2021-03-24T20:54:32.905 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-29 zones={test-zone2, test-zone1} 2021-03-24T20:54:32.914 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:54:32.963 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:54:33.024 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-29 zones={test-zone2, test-zone1} 2021-03-24T20:54:33.025 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-29 zones={test-zone1, test-zone2} 2021-03-24T20:54:33.035 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:54:33.083 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:54:33.144 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-29 zones={test-zone1, test-zone2} 2021-03-24T20:54:33.144 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-30 zones={test-zone2, test-zone1} 2021-03-24T20:54:33.154 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:54:33.199 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:54:33.256 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-30 zones={test-zone2, test-zone1} 2021-03-24T20:54:33.256 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-30 zones={test-zone1, test-zone2} 2021-03-24T20:54:33.264 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:54:33.315 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:54:33.372 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-30 zones={test-zone1, test-zone2} 2021-03-24T20:54:33.373 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-31 zones={test-zone2, test-zone1} 2021-03-24T20:54:33.382 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:54:33.431 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:54:33.492 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-31 zones={test-zone2, test-zone1} 2021-03-24T20:54:33.493 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-31 zones={test-zone1, test-zone2} 2021-03-24T20:54:33.502 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:54:33.550 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:54:33.616 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-31 zones={test-zone1, test-zone2} 2021-03-24T20:54:33.617 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-32 zones={test-zone2, test-zone1} 2021-03-24T20:54:33.627 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:54:33.675 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:54:33.737 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-32 zones={test-zone2, test-zone1} 2021-03-24T20:54:33.738 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-32 zones={test-zone1, test-zone2} 2021-03-24T20:54:33.747 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T20:54:33.795 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T20:54:33.861 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-32 zones={test-zone1, test-zone2} 2021-03-24T20:54:33.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'ded1abf8-1b45-4199-a116-504fb1354378', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:33.862 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'ded1abf8-1b45-4199-a116-504fb1354378', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:33.863 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period ded1abf8-1b45-4199-a116-504fb1354378 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:33.900 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:33.901 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:34.098 INFO:teuthology.orchestra.run.smithi052.stdout:[{"id":"1_1616618820.145235_26.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-24T20:47:00.145235727Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X6UefQMQBMk6T-KdROwDSmH","ver":1},"status":{"status":"write"}}},{"id":"1_1616618820.151371_27.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-24T20:47:00.151371959Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X6UefQMQBMk6T-KdROwDSmH","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618940.525007_30.1","section":"bucket.instance","name":"gtvmdf-1:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.1","timestamp":"2021-03-24T20:49:00.525007206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tgjzzXByXYIhUertDxJ3MjG","ver":1},"status":{"status":"write"}}},{"id":"1_1616618940.540912_31.1","section":"bucket.instance","name":"gtvmdf-1:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.1","timestamp":"2021-03-24T20:49:00.540912897Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tgjzzXByXYIhUertDxJ3MjG","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618940.542501_32.1","section":"bucket","name":"gtvmdf-1","timestamp":"2021-03-24T20:49:00.542501378Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xZ_E0OM_nU3o_x6hJ1NBmwF","ver":1},"status":{"status":"write"}}},{"id":"1_1616618940.546034_33.1","section":"bucket","name":"gtvmdf-1","timestamp":"2021-03-24T20:49:00.546034737Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xZ_E0OM_nU3o_x6hJ1NBmwF","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618940.562912_34.1","section":"bucket.instance","name":"gtvmdf-2:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.2","timestamp":"2021-03-24T20:49:00.562912585Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r-IHYx4xdxs8MH2QannkeLk","ver":1},"status":{"status":"write"}}},{"id":"1_1616618940.567183_35.1","section":"bucket.instance","name":"gtvmdf-2:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.2","timestamp":"2021-03-24T20:49:00.567183856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r-IHYx4xdxs8MH2QannkeLk","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618940.568689_36.1","section":"bucket","name":"gtvmdf-2","timestamp":"2021-03-24T20:49:00.568689460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jxLjNlFghqKuqp9MSw-Cwtb","ver":1},"status":{"status":"write"}}},{"id":"1_1616618940.572891_37.1","section":"bucket","name":"gtvmdf-2","timestamp":"2021-03-24T20:49:00.572891337Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jxLjNlFghqKuqp9MSw-Cwtb","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618950.394960_38.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.3","timestamp":"2021-03-24T20:49:10.394960181Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WNWA9V0cZEGDVBG0fH9bWv4","ver":1},"status":{"status":"write"}}},{"id":"1_1616618950.400191_39.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.3","timestamp":"2021-03-24T20:49:10.400191984Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WNWA9V0cZEGDVBG0fH9bWv4","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618950.402028_40.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:10.402028094Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LKSVbZjQiFBgTZb5X0XuzWc","ver":1},"status":{"status":"write"}}},{"id":"1_1616618950.406129_41.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:10.406129774Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LKSVbZjQiFBgTZb5X0XuzWc","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618950.421692_42.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.4","timestamp":"2021-03-24T20:49:10.421692289Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ib9mDs69W5PPcxzE8BrWHAf","ver":1},"status":{"status":"write"}}},{"id":"1_1616618950.426290_43.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.4","timestamp":"2021-03-24T20:49:10.426290203Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ib9mDs69W5PPcxzE8BrWHAf","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618950.428035_44.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:10.428035108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbhFBHo_oGklTk1VOtTLrbX","ver":1},"status":{"status":"write"}}},{"id":"1_1616618950.432415_45.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:10.432415906Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbhFBHo_oGklTk1VOtTLrbX","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618956.535316_46.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.5","timestamp":"2021-03-24T20:49:16.535316144Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cjMnKx2EFjhHoXEtD9Ao2Z3","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.541236_47.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.5","timestamp":"2021-03-24T20:49:16.541236512Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cjMnKx2EFjhHoXEtD9Ao2Z3","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618956.542678_48.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:16.542678387Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yESViFgr9Rbub7DYZK_HyfE","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.545516_49.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:16.545516233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yESViFgr9Rbub7DYZK_HyfE","ver":1},"status":{"status":"abort"}}},{"id":"1_1616618956.565500_50.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.6","timestamp":"2021-03-24T20:49:16.565500905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2bUZj_Da_6fOB3pOlD5P2y9","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.570055_51.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.6","timestamp":"2021-03-24T20:49:16.570055399Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2bUZj_Da_6fOB3pOlD5P2y9","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618956.571963_52.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:16.571963256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3fzyRfY3Cujq3-aYLKy_nKM","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.575616_53.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:16.575616943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3fzyRfY3Cujq3-aYLKy_nKM","ver":1},"status":{"status":"abort"}}},{"id":"1_1616618956.598421_54.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.7","timestamp":"2021-03-24T20:49:16.598421500Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ybNuFDA4bFl1DSj1nk48-Ml","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.603481_55.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.7","timestamp":"2021-03-24T20:49:16.603481230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ybNuFDA4bFl1DSj1nk48-Ml","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618956.605148_56.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:16.605148162Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Obi7HCPLocdgyHzF-nOiBpd","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.608201_57.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:16.608201871Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Obi7HCPLocdgyHzF-nOiBpd","ver":1},"status":{"status":"abort"}}},{"id":"1_1616618956.649240_58.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.8","timestamp":"2021-03-24T20:49:16.649240071Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pbPLxcJ1jzhAS9F523ewOAS","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.654691_59.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.8","timestamp":"2021-03-24T20:49:16.654691122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pbPLxcJ1jzhAS9F523ewOAS","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618956.656556_60.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:16.656556425Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kmWCjULi2Urt4nuHP9Fgda6","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.659390_61.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:16.659390794Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kmWCjULi2Urt4nuHP9Fgda6","ver":1},"status":{"status":"abort"}}},{"id":"1_1616618962.840164_62.1","section":"bucket.instance","name":"gtvmdf-5:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.9","timestamp":"2021-03-24T20:49:22.840164167Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vDfxx2c7LZmmfV2SYZvv8dT","ver":1},"status":{"status":"write"}}},{"id":"1_1616618962.846281_63.1","section":"bucket.instance","name":"gtvmdf-5:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.9","timestamp":"2021-03-24T20:49:22.846281903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vDfxx2c7LZmmfV2SYZvv8dT","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618962.848251_64.1","section":"bucket","name":"gtvmdf-5","timestamp":"2021-03-24T20:49:22.848251345Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":1},"status":{"status":"write"}}},{"id":"1_1616618962.852229_65.1","section":"bucket","name":"gtvmdf-5","timestamp":"2021-03-24T20:49:22.852229351Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618962.868458_66.1","section":"bucket.instance","name":"gtvmdf-6:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.10","timestamp":"2021-03-24T20:49:22.868458363Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Eu4nsv2VvM3EbZWZoASFgNS","ver":1},"status":{"status":"write"}}},{"id":"1_1616618962.873247_67.1","section":"bucket.instance","name":"gtvmdf-6:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.10","timestamp":"2021-03-24T20:49:22.873247875Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Eu4nsv2VvM3EbZWZoASFgNS","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618962.874924_68.1","section":"bucket","name":"gtvmdf-6","timestamp":"2021-03-24T20:49:22.874924380Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":1},"status":{"status":"write"}}},{"id":"1_1616618962.879387_69.1","section":"bucket","name":"gtvmdf-6","timestamp":"2021-03-24T20:49:22.879387934Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618969.082901_70.1","section":"bucket","name":"gtvmdf-5","timestamp":"2021-03-24T20:49:29.082901809Z","data":{"read_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":1},"write_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":2},"status":{"status":"remove"}}},{"id":"1_1616618969.088732_71.1","section":"bucket","name":"gtvmdf-5","timestamp":"2021-03-24T20:49:29.088732072Z","data":{"read_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":1},"write_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":2},"status":{"status":"complete"}}},{"id":"1_1616618969.124397_72.1","section":"bucket","name":"gtvmdf-6","timestamp":"2021-03-24T20:49:29.124397201Z","data":{"read_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":1},"write_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":2},"status":{"status":"remove"}}},{"id":"1_1616618969.129804_73.1","section":"bucket","name":"gtvmdf-6","timestamp":"2021-03-24T20:49:29.129804567Z","data":{"read_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":1},"write_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":2},"status":{"status":"complete"}}},{"id":"1_1616618975.259328_74.1","section":"bucket.instance","name":"gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11","timestamp":"2021-03-24T20:49:35.259328006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sxZH7m-QFlmDpKA8AgtzhPF","ver":1},"status":{"status":"write"}}},{"id":"1_1616618975.264835_75.1","section":"bucket.instance","name":"gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11","timestamp":"2021-03-24T20:49:35.264835737Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sxZH7m-QFlmDpKA8AgtzhPF","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618975.266208_76.1","section":"bucket","name":"gtvmdf-7","timestamp":"2021-03-24T20:49:35.266208098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zz5yVnTJtCVXtuxON_a8ehY","ver":1},"status":{"status":"write"}}},{"id":"1_1616618975.270002_77.1","section":"bucket","name":"gtvmdf-7","timestamp":"2021-03-24T20:49:35.270002818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zz5yVnTJtCVXtuxON_a8ehY","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618975.284878_78.1","section":"bucket.instance","name":"gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12","timestamp":"2021-03-24T20:49:35.284878435Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KMZLUo4l0O9JD4Hr04kuPbL","ver":1},"status":{"status":"write"}}},{"id":"1_1616618975.289559_79.1","section":"bucket.instance","name":"gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12","timestamp":"2021-03-24T20:49:35.289559510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KMZLUo4l0O9JD4Hr04kuPbL","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618975.291378_80.1","section":"bucket","name":"gtvmdf-8","timestamp":"2021-03-24T20:49:35.291378616Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jy6J99ZlUqP_RnQt-0vqafJ","ver":1},"status":{"status":"write"}}},{"id":"1_1616618975.295785_81.1","section":"bucket","name":"gtvmdf-8","timestamp":"2021-03-24T20:49:35.295785394Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jy6J99ZlUqP_RnQt-0vqafJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618986.091302_82.1","section":"bucket.instance","name":"gtvmdf-9:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.13","timestamp":"2021-03-24T20:49:46.091302407Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-Vo7HKGnWkmFy0AVuOrx8vD","ver":1},"status":{"status":"write"}}},{"id":"1_1616618986.095790_83.1","section":"bucket.instance","name":"gtvmdf-9:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.13","timestamp":"2021-03-24T20:49:46.095790175Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-Vo7HKGnWkmFy0AVuOrx8vD","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618986.097257_84.1","section":"bucket","name":"gtvmdf-9","timestamp":"2021-03-24T20:49:46.097257755Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oylq7sQHahJX4um-hkauus5","ver":1},"status":{"status":"write"}}},{"id":"1_1616618986.101275_85.1","section":"bucket","name":"gtvmdf-9","timestamp":"2021-03-24T20:49:46.101275951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oylq7sQHahJX4um-hkauus5","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618986.116223_86.1","section":"bucket.instance","name":"gtvmdf-10:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.14","timestamp":"2021-03-24T20:49:46.116223321Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lPn06j2Mk7rBKrns9S4gsnh","ver":1},"status":{"status":"write"}}},{"id":"1_1616618986.121070_87.1","section":"bucket.instance","name":"gtvmdf-10:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.14","timestamp":"2021-03-24T20:49:46.121070252Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lPn06j2Mk7rBKrns9S4gsnh","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618986.123005_88.1","section":"bucket","name":"gtvmdf-10","timestamp":"2021-03-24T20:49:46.123005877Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9l4AizNcZyFC7-zC0CB1TeJ","ver":1},"status":{"status":"write"}}},{"id":"1_1616618986.127919_89.1","section":"bucket","name":"gtvmdf-10","timestamp":"2021-03-24T20:49:46.127919319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9l4AizNcZyFC7-zC0CB1TeJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619007.041460_90.1","section":"bucket.instance","name":"gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15","timestamp":"2021-03-24T20:50:07.041460956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":1},"status":{"status":"write"}}},{"id":"1_1616619007.047026_91.1","section":"bucket.instance","name":"gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15","timestamp":"2021-03-24T20:50:07.047026910Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619007.048478_92.1","section":"bucket","name":"gtvmdf-11","timestamp":"2021-03-24T20:50:07.048478376Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AgLAtS2oyLPaHP6unNPtmVC","ver":1},"status":{"status":"write"}}},{"id":"1_1616619007.052429_93.1","section":"bucket","name":"gtvmdf-11","timestamp":"2021-03-24T20:50:07.052429307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AgLAtS2oyLPaHP6unNPtmVC","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619007.067245_94.1","section":"bucket.instance","name":"gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16","timestamp":"2021-03-24T20:50:07.067245681Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":1},"status":{"status":"write"}}},{"id":"1_1616619007.072112_95.1","section":"bucket.instance","name":"gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16","timestamp":"2021-03-24T20:50:07.072112238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619007.074206_96.1","section":"bucket","name":"gtvmdf-12","timestamp":"2021-03-24T20:50:07.074206526Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DQRLU6jS7zHVvjx0sgi1qPV","ver":1},"status":{"status":"write"}}},{"id":"1_1616619007.079276_97.1","section":"bucket","name":"gtvmdf-12","timestamp":"2021-03-24T20:50:07.079276614Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DQRLU6jS7zHVvjx0sgi1qPV","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619007.103931_98.1","section":"bucket.instance","name":"gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15","timestamp":"2021-03-24T20:50:07.103931395Z","data":{"read_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":1},"write_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":2},"status":{"status":"write"}}},{"id":"1_1616619007.110653_99.1","section":"bucket.instance","name":"gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15","timestamp":"2021-03-24T20:50:07.110653160Z","data":{"read_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":1},"write_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619007.116106_100.1","section":"bucket.instance","name":"gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16","timestamp":"2021-03-24T20:50:07.116106696Z","data":{"read_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":1},"write_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":2},"status":{"status":"write"}}},{"id":"1_1616619007.123420_101.1","section":"bucket.instance","name":"gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16","timestamp":"2021-03-24T20:50:07.123420937Z","data":{"read_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":1},"write_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619035.929342_102.1","section":"bucket.instance","name":"gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17","timestamp":"2021-03-24T20:50:35.929342176Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":1},"status":{"status":"write"}}},{"id":"1_1616619035.934528_103.1","section":"bucket.instance","name":"gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17","timestamp":"2021-03-24T20:50:35.934528828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619035.936547_104.1","section":"bucket","name":"gtvmdf-13","timestamp":"2021-03-24T20:50:35.936547600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTs-jZYrEVVtkrfK3uz3f0b","ver":1},"status":{"status":"write"}}},{"id":"1_1616619035.940636_105.1","section":"bucket","name":"gtvmdf-13","timestamp":"2021-03-24T20:50:35.940636057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTs-jZYrEVVtkrfK3uz3f0b","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619035.945564_106.1","section":"bucket.instance","name":"gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17","timestamp":"2021-03-24T20:50:35.945564609Z","data":{"read_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":1},"write_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":2},"status":{"status":"write"}}},{"id":"1_1616619035.950831_107.1","section":"bucket.instance","name":"gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17","timestamp":"2021-03-24T20:50:35.950831213Z","data":{"read_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":1},"write_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619059.395270_108.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:50:59.395270616Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":1},"status":{"status":"write"}}},{"id":"1_1616619059.401521_109.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:50:59.401521462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619059.402951_110.1","section":"bucket","name":"gtvmdf-14","timestamp":"2021-03-24T20:50:59.402951206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ByGa8IEaJecc2xys3DlWnyS","ver":1},"status":{"status":"write"}}},{"id":"1_1616619059.407165_111.1","section":"bucket","name":"gtvmdf-14","timestamp":"2021-03-24T20:50:59.407165083Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ByGa8IEaJecc2xys3DlWnyS","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619072.516314_112.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:51:12.516314795Z","data":{"read_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":1},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":2},"status":{"status":"write"}}},{"id":"1_1616619072.523480_113.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:51:12.523480675Z","data":{"read_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":1},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619085.699597_114.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:51:25.699597319Z","data":{"read_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":2},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":3},"status":{"status":"write"}}},{"id":"1_1616619085.706880_115.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:51:25.706880905Z","data":{"read_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":2},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":3},"status":{"status":"complete"}}},{"id":"1_1616619098.934086_116.1","section":"bucket.instance","name":"gtvmdf-15:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.19","timestamp":"2021-03-24T20:51:38.934086328Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":1},"status":{"status":"write"}}},{"id":"1_1616619098.939240_117.1","section":"bucket.instance","name":"gtvmdf-15:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.19","timestamp":"2021-03-24T20:51:38.939240913Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619098.941108_118.1","section":"bucket","name":"gtvmdf-15","timestamp":"2021-03-24T20:51:38.941108427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PNuc2zTOz6GyFXbZw-fIBWb","ver":1},"status":{"status":"write"}}},{"id":"1_1616619098.945494_119.1","section":"bucket","name":"gtvmdf-15","timestamp":"2021-03-24T20:51:38.945494713Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PNuc2zTOz6GyFXbZw-fIBWb","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619098.961611_120.1","section":"bucket.instance","name":"gtvmdf-16:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.20","timestamp":"2021-03-24T20:51:38.961611938Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":1},"status":{"status":"write"}}},{"id":"1_1616619098.966402_121.1","section":"bucket.instance","name":"gtvmdf-16:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.20","timestamp":"2021-03-24T20:51:38.966402125Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619098.968020_122.1","section":"bucket","name":"gtvmdf-16","timestamp":"2021-03-24T20:51:38.968020620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_31UMZdp1WoWzvotn7Nsm__Y","ver":1},"status":{"status":"write"}}},{"id":"1_1616619098.972216_123.1","section":"bucket","name":"gtvmdf-16","timestamp":"2021-03-24T20:51:38.972216159Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_31UMZdp1WoWzvotn7Nsm__Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619098.996018_124.1","section":"bucket.instance","name":"gtvmdf-15:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.19","timestamp":"2021-03-24T20:51:38.996018519Z","data":{"read_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":1},"write_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":2},"status":{"status":"write"}}},{"id":"1_1616619099.001750_125.1","section":"bucket.instance","name":"gtvmdf-15:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.19","timestamp":"2021-03-24T20:51:39.001750396Z","data":{"read_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":1},"write_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619099.006889_126.1","section":"bucket.instance","name":"gtvmdf-16:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.20","timestamp":"2021-03-24T20:51:39.006889838Z","data":{"read_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":1},"write_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":2},"status":{"status":"write"}}},{"id":"1_1616619099.012600_127.1","section":"bucket.instance","name":"gtvmdf-16:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.20","timestamp":"2021-03-24T20:51:39.012600803Z","data":{"read_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":1},"write_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619114.027971_128.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.027971254Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":1},"status":{"status":"write"}}},{"id":"1_1616619114.032794_129.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.032794703Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619114.034888_130.1","section":"bucket","name":"gtvmdf-17","timestamp":"2021-03-24T20:51:54.034888486Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R3kKWlMS5jbJmCin0m0U6-V","ver":1},"status":{"status":"write"}}},{"id":"1_1616619114.039578_131.1","section":"bucket","name":"gtvmdf-17","timestamp":"2021-03-24T20:51:54.039578229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R3kKWlMS5jbJmCin0m0U6-V","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619114.057862_132.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.057862668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":1},"status":{"status":"write"}}},{"id":"1_1616619114.062774_133.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.062774722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619114.064381_134.1","section":"bucket","name":"gtvmdf-18","timestamp":"2021-03-24T20:51:54.064381248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pY0JjBJHuqzFvxk29Qv5iEk","ver":1},"status":{"status":"write"}}},{"id":"1_1616619114.068519_135.1","section":"bucket","name":"gtvmdf-18","timestamp":"2021-03-24T20:51:54.068519964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pY0JjBJHuqzFvxk29Qv5iEk","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619114.091214_136.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.091214680Z","data":{"read_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":1},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":2},"status":{"status":"write"}}},{"id":"1_1616619114.097005_137.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.097005230Z","data":{"read_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":1},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619114.101388_138.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.101388916Z","data":{"read_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":2},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":3},"status":{"status":"write"}}},{"id":"1_1616619114.107752_139.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.107752477Z","data":{"read_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":2},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":3},"status":{"status":"complete"}}},{"id":"1_1616619114.112716_140.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.112716205Z","data":{"read_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":1},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":2},"status":{"status":"write"}}},{"id":"1_1616619114.118374_141.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.118374960Z","data":{"read_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":1},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619114.127787_142.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.127787742Z","data":{"read_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":2},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":3},"status":{"status":"write"}}},{"id":"1_1616619114.134086_143.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.134086520Z","data":{"read_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":2},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":3},"status":{"status":"complete"}}},{"id":"1_1616619128.926611_144.1","section":"bucket.instance","name":"gtvmdf-19:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.23","timestamp":"2021-03-24T20:52:08.926611189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":1},"status":{"status":"write"}}},{"id":"1_1616619128.931253_145.1","section":"bucket.instance","name":"gtvmdf-19:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.23","timestamp":"2021-03-24T20:52:08.931253162Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619128.932758_146.1","section":"bucket","name":"gtvmdf-19","timestamp":"2021-03-24T20:52:08.932758889Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Je5wveQUSZ4tEH7MFOjdkdc","ver":1},"status":{"status":"write"}}},{"id":"1_1616619128.936408_147.1","section":"bucket","name":"gtvmdf-19","timestamp":"2021-03-24T20:52:08.936408958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Je5wveQUSZ4tEH7MFOjdkdc","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619128.951841_148.1","section":"bucket.instance","name":"gtvmdf-20:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.24","timestamp":"2021-03-24T20:52:08.951841872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":1},"status":{"status":"write"}}},{"id":"1_1616619128.956923_149.1","section":"bucket.instance","name":"gtvmdf-20:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.24","timestamp":"2021-03-24T20:52:08.956923847Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619128.958767_150.1","section":"bucket","name":"gtvmdf-20","timestamp":"2021-03-24T20:52:08.958767229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8N5Cfahn8cjfgL9uoeNSadH","ver":1},"status":{"status":"write"}}},{"id":"1_1616619128.963465_151.1","section":"bucket","name":"gtvmdf-20","timestamp":"2021-03-24T20:52:08.963465691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8N5Cfahn8cjfgL9uoeNSadH","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619128.986202_152.1","section":"bucket.instance","name":"gtvmdf-19:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.23","timestamp":"2021-03-24T20:52:08.986202010Z","data":{"read_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":1},"write_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":2},"status":{"status":"write"}}},{"id":"1_1616619128.992016_153.1","section":"bucket.instance","name":"gtvmdf-19:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.23","timestamp":"2021-03-24T20:52:08.992016781Z","data":{"read_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":1},"write_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619129.046345_154.1","section":"bucket.instance","name":"gtvmdf-20:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.24","timestamp":"2021-03-24T20:52:09.046345165Z","data":{"read_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":1},"write_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":2},"status":{"status":"write"}}},{"id":"1_1616619129.054539_155.1","section":"bucket.instance","name":"gtvmdf-20:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.24","timestamp":"2021-03-24T20:52:09.054539077Z","data":{"read_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":1},"write_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619129.123945_156.1","section":"bucket.instance","name":"gtvmdf-21:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.25","timestamp":"2021-03-24T20:52:09.123945194Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.128767_157.1","section":"bucket.instance","name":"gtvmdf-21:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.25","timestamp":"2021-03-24T20:52:09.128767956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.130274_158.1","section":"bucket","name":"gtvmdf-21","timestamp":"2021-03-24T20:52:09.130274136Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8UZ17cnIdx1ZJ8Cp87nvsOz","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.133740_159.1","section":"bucket","name":"gtvmdf-21","timestamp":"2021-03-24T20:52:09.133740102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8UZ17cnIdx1ZJ8Cp87nvsOz","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.149337_160.1","section":"bucket.instance","name":"gtvmdf-22:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.26","timestamp":"2021-03-24T20:52:09.149337081Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.154507_161.1","section":"bucket.instance","name":"gtvmdf-22:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.26","timestamp":"2021-03-24T20:52:09.154507107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.156397_162.1","section":"bucket","name":"gtvmdf-22","timestamp":"2021-03-24T20:52:09.156397349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__6dVONzzpn69bgtpbFZsSAa","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.161099_163.1","section":"bucket","name":"gtvmdf-22","timestamp":"2021-03-24T20:52:09.161099316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__6dVONzzpn69bgtpbFZsSAa","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.185839_164.1","section":"bucket.instance","name":"gtvmdf-21:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.25","timestamp":"2021-03-24T20:52:09.185839337Z","data":{"read_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":1},"write_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":2},"status":{"status":"write"}}},{"id":"1_1616619129.192122_165.1","section":"bucket.instance","name":"gtvmdf-21:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.25","timestamp":"2021-03-24T20:52:09.192122581Z","data":{"read_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":1},"write_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619129.242797_166.1","section":"bucket.instance","name":"gtvmdf-22:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.26","timestamp":"2021-03-24T20:52:09.242797437Z","data":{"read_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":1},"write_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":2},"status":{"status":"write"}}},{"id":"1_1616619129.250113_167.1","section":"bucket.instance","name":"gtvmdf-22:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.26","timestamp":"2021-03-24T20:52:09.250113393Z","data":{"read_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":1},"write_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619129.315702_168.1","section":"bucket.instance","name":"gtvmdf-23:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.27","timestamp":"2021-03-24T20:52:09.315702943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.320009_169.1","section":"bucket.instance","name":"gtvmdf-23:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.27","timestamp":"2021-03-24T20:52:09.320009574Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.321588_170.1","section":"bucket","name":"gtvmdf-23","timestamp":"2021-03-24T20:52:09.321588220Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1TrqWqWCLiV25-ESjq7E271","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.325364_171.1","section":"bucket","name":"gtvmdf-23","timestamp":"2021-03-24T20:52:09.325364400Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1TrqWqWCLiV25-ESjq7E271","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.340715_172.1","section":"bucket.instance","name":"gtvmdf-24:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.28","timestamp":"2021-03-24T20:52:09.340715588Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.345437_173.1","section":"bucket.instance","name":"gtvmdf-24:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.28","timestamp":"2021-03-24T20:52:09.345437875Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.347357_174.1","section":"bucket","name":"gtvmdf-24","timestamp":"2021-03-24T20:52:09.347357319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6FJJvGEakJfW6YWu5fy1V6x","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.351703_175.1","section":"bucket","name":"gtvmdf-24","timestamp":"2021-03-24T20:52:09.351703588Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6FJJvGEakJfW6YWu5fy1V6x","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.377737_176.1","section":"bucket.instance","name":"gtvmdf-23:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.27","timestamp":"2021-03-24T20:52:09.377737881Z","data":{"read_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":1},"write_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":2},"status":{"status":"write"}}},{"id":"1_1616619129.383378_177.1","section":"bucket.instance","name":"gtvmdf-23:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.27","timestamp":"2021-03-24T20:52:09.383378032Z","data":{"read_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":1},"write_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619130.433454_178.1","section":"bucket.instance","name":"gtvmdf-24:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.28","timestamp":"2021-03-24T20:52:10.433454675Z","data":{"read_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":1},"write_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":2},"status":{"status":"write"}}},{"id":"1_1616619130.441232_179.1","section":"bucket.instance","name":"gtvmdf-24:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.28","timestamp":"2021-03-24T20:52:10.441232968Z","data":{"read_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":1},"write_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619130.506282_180.1","section":"bucket.instance","name":"gtvmdf-25:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.29","timestamp":"2021-03-24T20:52:10.506282155Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fXX3g3kQ16Gph9Q2cFs-vHR","ver":1},"status":{"status":"write"}}},{"id":"1_1616619130.510800_181.1","section":"bucket.instance","name":"gtvmdf-25:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.29","timestamp":"2021-03-24T20:52:10.510800522Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fXX3g3kQ16Gph9Q2cFs-vHR","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619130.512288_182.1","section":"bucket","name":"gtvmdf-25","timestamp":"2021-03-24T20:52:10.512288206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hEyZngLUiA6ZyGw-3jS1LKQ","ver":1},"status":{"status":"write"}}},{"id":"1_1616619130.516322_183.1","section":"bucket","name":"gtvmdf-25","timestamp":"2021-03-24T20:52:10.516322307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hEyZngLUiA6ZyGw-3jS1LKQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619130.532853_184.1","section":"bucket.instance","name":"gtvmdf-26:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.30","timestamp":"2021-03-24T20:52:10.532853024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tS4SKRvZjZho99JH5a6148C","ver":1},"status":{"status":"write"}}},{"id":"1_1616619130.538378_185.1","section":"bucket.instance","name":"gtvmdf-26:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.30","timestamp":"2021-03-24T20:52:10.538378369Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tS4SKRvZjZho99JH5a6148C","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619130.539793_186.1","section":"bucket","name":"gtvmdf-26","timestamp":"2021-03-24T20:52:10.539793096Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZxL9Rfkew_YALii_N7IEBqX","ver":1},"status":{"status":"write"}}},{"id":"1_1616619130.543652_187.1","section":"bucket","name":"gtvmdf-26","timestamp":"2021-03-24T20:52:10.543652869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZxL9Rfkew_YALii_N7IEBqX","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619136.909575_188.1","section":"bucket.instance","name":"gtvmdf-27:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.31","timestamp":"2021-03-24T20:52:16.909575843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_d0c6LC8vx4WUhi8kjAeTuGR","ver":1},"status":{"status":"write"}}},{"id":"1_1616619136.913840_189.1","section":"bucket.instance","name":"gtvmdf-27:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.31","timestamp":"2021-03-24T20:52:16.913840638Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_d0c6LC8vx4WUhi8kjAeTuGR","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619136.915687_190.1","section":"bucket","name":"gtvmdf-27","timestamp":"2021-03-24T20:52:16.915687135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Macs6T62O19upzCNiUpxIIS","ver":1},"status":{"status":"write"}}},{"id":"1_1616619136.920010_191.1","section":"bucket","name":"gtvmdf-27","timestamp":"2021-03-24T20:52:16.920010356Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Macs6T62O19upzCNiUpxIIS","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619136.935694_192.1","section":"bucket.instance","name":"gtvmdf-28:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.32","timestamp":"2021-03-24T20:52:16.935694455Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PPoDHP_LH5U_Pp0M6wWsnd7","ver":1},"status":{"status":"write"}}},{"id":"1_1616619136.940603_193.1","section":"bucket.instance","name":"gtvmdf-28:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.32","timestamp":"2021-03-24T20:52:16.940603762Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PPoDHP_LH5U_Pp0M6wWsnd7","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619136.942415_194.1","section":"bucket","name":"gtvmdf-28","timestamp":"2021-03-24T20:52:16.942415111Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H0XNwOTOzmyG2T2RQga4hg2","ver":1},"status":{"status":"write"}}},{"id":"1_1616619136.946453_195.1","section":"bucket","name":"gtvmdf-28","timestamp":"2021-03-24T20:52:16.946453255Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H0XNwOTOzmyG2T2RQga4hg2","ver":1},"status":{"status":"complete"}}}] 2021-03-24T20:54:34.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'ded1abf8-1b45-4199-a116-504fb1354378', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:34.100 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'ded1abf8-1b45-4199-a116-504fb1354378', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:34.101 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period ded1abf8-1b45-4199-a116-504fb1354378 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:34.139 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:34.139 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:34.332 INFO:teuthology.orchestra.run.smithi063.stdout:[{"id":"1_1616618820.145235_26.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-24T20:47:00.145235727Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X6UefQMQBMk6T-KdROwDSmH","ver":1},"status":{"status":"write"}}},{"id":"1_1616618820.151371_27.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-24T20:47:00.151371959Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X6UefQMQBMk6T-KdROwDSmH","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618940.525007_30.1","section":"bucket.instance","name":"gtvmdf-1:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.1","timestamp":"2021-03-24T20:49:00.525007206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tgjzzXByXYIhUertDxJ3MjG","ver":1},"status":{"status":"write"}}},{"id":"1_1616618940.540912_31.1","section":"bucket.instance","name":"gtvmdf-1:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.1","timestamp":"2021-03-24T20:49:00.540912897Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tgjzzXByXYIhUertDxJ3MjG","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618940.542501_32.1","section":"bucket","name":"gtvmdf-1","timestamp":"2021-03-24T20:49:00.542501378Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xZ_E0OM_nU3o_x6hJ1NBmwF","ver":1},"status":{"status":"write"}}},{"id":"1_1616618940.546034_33.1","section":"bucket","name":"gtvmdf-1","timestamp":"2021-03-24T20:49:00.546034737Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xZ_E0OM_nU3o_x6hJ1NBmwF","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618940.562912_34.1","section":"bucket.instance","name":"gtvmdf-2:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.2","timestamp":"2021-03-24T20:49:00.562912585Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r-IHYx4xdxs8MH2QannkeLk","ver":1},"status":{"status":"write"}}},{"id":"1_1616618940.567183_35.1","section":"bucket.instance","name":"gtvmdf-2:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.2","timestamp":"2021-03-24T20:49:00.567183856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r-IHYx4xdxs8MH2QannkeLk","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618940.568689_36.1","section":"bucket","name":"gtvmdf-2","timestamp":"2021-03-24T20:49:00.568689460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jxLjNlFghqKuqp9MSw-Cwtb","ver":1},"status":{"status":"write"}}},{"id":"1_1616618940.572891_37.1","section":"bucket","name":"gtvmdf-2","timestamp":"2021-03-24T20:49:00.572891337Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jxLjNlFghqKuqp9MSw-Cwtb","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618950.394960_38.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.3","timestamp":"2021-03-24T20:49:10.394960181Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WNWA9V0cZEGDVBG0fH9bWv4","ver":1},"status":{"status":"write"}}},{"id":"1_1616618950.400191_39.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.3","timestamp":"2021-03-24T20:49:10.400191984Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WNWA9V0cZEGDVBG0fH9bWv4","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618950.402028_40.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:10.402028094Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LKSVbZjQiFBgTZb5X0XuzWc","ver":1},"status":{"status":"write"}}},{"id":"1_1616618950.406129_41.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:10.406129774Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LKSVbZjQiFBgTZb5X0XuzWc","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618950.421692_42.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.4","timestamp":"2021-03-24T20:49:10.421692289Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ib9mDs69W5PPcxzE8BrWHAf","ver":1},"status":{"status":"write"}}},{"id":"1_1616618950.426290_43.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.4","timestamp":"2021-03-24T20:49:10.426290203Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ib9mDs69W5PPcxzE8BrWHAf","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618950.428035_44.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:10.428035108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbhFBHo_oGklTk1VOtTLrbX","ver":1},"status":{"status":"write"}}},{"id":"1_1616618950.432415_45.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:10.432415906Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbhFBHo_oGklTk1VOtTLrbX","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618956.535316_46.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.5","timestamp":"2021-03-24T20:49:16.535316144Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cjMnKx2EFjhHoXEtD9Ao2Z3","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.541236_47.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.5","timestamp":"2021-03-24T20:49:16.541236512Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cjMnKx2EFjhHoXEtD9Ao2Z3","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618956.542678_48.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:16.542678387Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yESViFgr9Rbub7DYZK_HyfE","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.545516_49.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:16.545516233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yESViFgr9Rbub7DYZK_HyfE","ver":1},"status":{"status":"abort"}}},{"id":"1_1616618956.565500_50.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.6","timestamp":"2021-03-24T20:49:16.565500905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2bUZj_Da_6fOB3pOlD5P2y9","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.570055_51.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.6","timestamp":"2021-03-24T20:49:16.570055399Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2bUZj_Da_6fOB3pOlD5P2y9","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618956.571963_52.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:16.571963256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3fzyRfY3Cujq3-aYLKy_nKM","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.575616_53.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:16.575616943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3fzyRfY3Cujq3-aYLKy_nKM","ver":1},"status":{"status":"abort"}}},{"id":"1_1616618956.598421_54.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.7","timestamp":"2021-03-24T20:49:16.598421500Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ybNuFDA4bFl1DSj1nk48-Ml","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.603481_55.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.7","timestamp":"2021-03-24T20:49:16.603481230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ybNuFDA4bFl1DSj1nk48-Ml","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618956.605148_56.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:16.605148162Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Obi7HCPLocdgyHzF-nOiBpd","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.608201_57.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:16.608201871Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Obi7HCPLocdgyHzF-nOiBpd","ver":1},"status":{"status":"abort"}}},{"id":"1_1616618956.649240_58.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.8","timestamp":"2021-03-24T20:49:16.649240071Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pbPLxcJ1jzhAS9F523ewOAS","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.654691_59.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.8","timestamp":"2021-03-24T20:49:16.654691122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pbPLxcJ1jzhAS9F523ewOAS","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618956.656556_60.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:16.656556425Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kmWCjULi2Urt4nuHP9Fgda6","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.659390_61.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:16.659390794Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kmWCjULi2Urt4nuHP9Fgda6","ver":1},"status":{"status":"abort"}}},{"id":"1_1616618962.840164_62.1","section":"bucket.instance","name":"gtvmdf-5:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.9","timestamp":"2021-03-24T20:49:22.840164167Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vDfxx2c7LZmmfV2SYZvv8dT","ver":1},"status":{"status":"write"}}},{"id":"1_1616618962.846281_63.1","section":"bucket.instance","name":"gtvmdf-5:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.9","timestamp":"2021-03-24T20:49:22.846281903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vDfxx2c7LZmmfV2SYZvv8dT","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618962.848251_64.1","section":"bucket","name":"gtvmdf-5","timestamp":"2021-03-24T20:49:22.848251345Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":1},"status":{"status":"write"}}},{"id":"1_1616618962.852229_65.1","section":"bucket","name":"gtvmdf-5","timestamp":"2021-03-24T20:49:22.852229351Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618962.868458_66.1","section":"bucket.instance","name":"gtvmdf-6:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.10","timestamp":"2021-03-24T20:49:22.868458363Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Eu4nsv2VvM3EbZWZoASFgNS","ver":1},"status":{"status":"write"}}},{"id":"1_1616618962.873247_67.1","section":"bucket.instance","name":"gtvmdf-6:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.10","timestamp":"2021-03-24T20:49:22.873247875Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Eu4nsv2VvM3EbZWZoASFgNS","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618962.874924_68.1","section":"bucket","name":"gtvmdf-6","timestamp":"2021-03-24T20:49:22.874924380Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":1},"status":{"status":"write"}}},{"id":"1_1616618962.879387_69.1","section":"bucket","name":"gtvmdf-6","timestamp":"2021-03-24T20:49:22.879387934Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618969.082901_70.1","section":"bucket","name":"gtvmdf-5","timestamp":"2021-03-24T20:49:29.082901809Z","data":{"read_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":1},"write_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":2},"status":{"status":"remove"}}},{"id":"1_1616618969.088732_71.1","section":"bucket","name":"gtvmdf-5","timestamp":"2021-03-24T20:49:29.088732072Z","data":{"read_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":1},"write_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":2},"status":{"status":"complete"}}},{"id":"1_1616618969.124397_72.1","section":"bucket","name":"gtvmdf-6","timestamp":"2021-03-24T20:49:29.124397201Z","data":{"read_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":1},"write_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":2},"status":{"status":"remove"}}},{"id":"1_1616618969.129804_73.1","section":"bucket","name":"gtvmdf-6","timestamp":"2021-03-24T20:49:29.129804567Z","data":{"read_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":1},"write_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":2},"status":{"status":"complete"}}},{"id":"1_1616618975.259328_74.1","section":"bucket.instance","name":"gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11","timestamp":"2021-03-24T20:49:35.259328006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sxZH7m-QFlmDpKA8AgtzhPF","ver":1},"status":{"status":"write"}}},{"id":"1_1616618975.264835_75.1","section":"bucket.instance","name":"gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11","timestamp":"2021-03-24T20:49:35.264835737Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sxZH7m-QFlmDpKA8AgtzhPF","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618975.266208_76.1","section":"bucket","name":"gtvmdf-7","timestamp":"2021-03-24T20:49:35.266208098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zz5yVnTJtCVXtuxON_a8ehY","ver":1},"status":{"status":"write"}}},{"id":"1_1616618975.270002_77.1","section":"bucket","name":"gtvmdf-7","timestamp":"2021-03-24T20:49:35.270002818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zz5yVnTJtCVXtuxON_a8ehY","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618975.284878_78.1","section":"bucket.instance","name":"gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12","timestamp":"2021-03-24T20:49:35.284878435Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KMZLUo4l0O9JD4Hr04kuPbL","ver":1},"status":{"status":"write"}}},{"id":"1_1616618975.289559_79.1","section":"bucket.instance","name":"gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12","timestamp":"2021-03-24T20:49:35.289559510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KMZLUo4l0O9JD4Hr04kuPbL","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618975.291378_80.1","section":"bucket","name":"gtvmdf-8","timestamp":"2021-03-24T20:49:35.291378616Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jy6J99ZlUqP_RnQt-0vqafJ","ver":1},"status":{"status":"write"}}},{"id":"1_1616618975.295785_81.1","section":"bucket","name":"gtvmdf-8","timestamp":"2021-03-24T20:49:35.295785394Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jy6J99ZlUqP_RnQt-0vqafJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618986.091302_82.1","section":"bucket.instance","name":"gtvmdf-9:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.13","timestamp":"2021-03-24T20:49:46.091302407Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-Vo7HKGnWkmFy0AVuOrx8vD","ver":1},"status":{"status":"write"}}},{"id":"1_1616618986.095790_83.1","section":"bucket.instance","name":"gtvmdf-9:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.13","timestamp":"2021-03-24T20:49:46.095790175Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-Vo7HKGnWkmFy0AVuOrx8vD","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618986.097257_84.1","section":"bucket","name":"gtvmdf-9","timestamp":"2021-03-24T20:49:46.097257755Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oylq7sQHahJX4um-hkauus5","ver":1},"status":{"status":"write"}}},{"id":"1_1616618986.101275_85.1","section":"bucket","name":"gtvmdf-9","timestamp":"2021-03-24T20:49:46.101275951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oylq7sQHahJX4um-hkauus5","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618986.116223_86.1","section":"bucket.instance","name":"gtvmdf-10:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.14","timestamp":"2021-03-24T20:49:46.116223321Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lPn06j2Mk7rBKrns9S4gsnh","ver":1},"status":{"status":"write"}}},{"id":"1_1616618986.121070_87.1","section":"bucket.instance","name":"gtvmdf-10:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.14","timestamp":"2021-03-24T20:49:46.121070252Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lPn06j2Mk7rBKrns9S4gsnh","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618986.123005_88.1","section":"bucket","name":"gtvmdf-10","timestamp":"2021-03-24T20:49:46.123005877Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9l4AizNcZyFC7-zC0CB1TeJ","ver":1},"status":{"status":"write"}}},{"id":"1_1616618986.127919_89.1","section":"bucket","name":"gtvmdf-10","timestamp":"2021-03-24T20:49:46.127919319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9l4AizNcZyFC7-zC0CB1TeJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619007.041460_90.1","section":"bucket.instance","name":"gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15","timestamp":"2021-03-24T20:50:07.041460956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":1},"status":{"status":"write"}}},{"id":"1_1616619007.047026_91.1","section":"bucket.instance","name":"gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15","timestamp":"2021-03-24T20:50:07.047026910Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619007.048478_92.1","section":"bucket","name":"gtvmdf-11","timestamp":"2021-03-24T20:50:07.048478376Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AgLAtS2oyLPaHP6unNPtmVC","ver":1},"status":{"status":"write"}}},{"id":"1_1616619007.052429_93.1","section":"bucket","name":"gtvmdf-11","timestamp":"2021-03-24T20:50:07.052429307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AgLAtS2oyLPaHP6unNPtmVC","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619007.067245_94.1","section":"bucket.instance","name":"gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16","timestamp":"2021-03-24T20:50:07.067245681Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":1},"status":{"status":"write"}}},{"id":"1_1616619007.072112_95.1","section":"bucket.instance","name":"gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16","timestamp":"2021-03-24T20:50:07.072112238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619007.074206_96.1","section":"bucket","name":"gtvmdf-12","timestamp":"2021-03-24T20:50:07.074206526Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DQRLU6jS7zHVvjx0sgi1qPV","ver":1},"status":{"status":"write"}}},{"id":"1_1616619007.079276_97.1","section":"bucket","name":"gtvmdf-12","timestamp":"2021-03-24T20:50:07.079276614Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DQRLU6jS7zHVvjx0sgi1qPV","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619007.103931_98.1","section":"bucket.instance","name":"gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15","timestamp":"2021-03-24T20:50:07.103931395Z","data":{"read_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":1},"write_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":2},"status":{"status":"write"}}},{"id":"1_1616619007.110653_99.1","section":"bucket.instance","name":"gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15","timestamp":"2021-03-24T20:50:07.110653160Z","data":{"read_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":1},"write_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619007.116106_100.1","section":"bucket.instance","name":"gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16","timestamp":"2021-03-24T20:50:07.116106696Z","data":{"read_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":1},"write_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":2},"status":{"status":"write"}}},{"id":"1_1616619007.123420_101.1","section":"bucket.instance","name":"gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16","timestamp":"2021-03-24T20:50:07.123420937Z","data":{"read_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":1},"write_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619035.929342_102.1","section":"bucket.instance","name":"gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17","timestamp":"2021-03-24T20:50:35.929342176Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":1},"status":{"status":"write"}}},{"id":"1_1616619035.934528_103.1","section":"bucket.instance","name":"gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17","timestamp":"2021-03-24T20:50:35.934528828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619035.936547_104.1","section":"bucket","name":"gtvmdf-13","timestamp":"2021-03-24T20:50:35.936547600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTs-jZYrEVVtkrfK3uz3f0b","ver":1},"status":{"status":"write"}}},{"id":"1_1616619035.940636_105.1","section":"bucket","name":"gtvmdf-13","timestamp":"2021-03-24T20:50:35.940636057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTs-jZYrEVVtkrfK3uz3f0b","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619035.945564_106.1","section":"bucket.instance","name":"gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17","timestamp":"2021-03-24T20:50:35.945564609Z","data":{"read_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":1},"write_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":2},"status":{"status":"write"}}},{"id":"1_1616619035.950831_107.1","section":"bucket.instance","name":"gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17","timestamp":"2021-03-24T20:50:35.950831213Z","data":{"read_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":1},"write_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619059.395270_108.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:50:59.395270616Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":1},"status":{"status":"write"}}},{"id":"1_1616619059.401521_109.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:50:59.401521462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619059.402951_110.1","section":"bucket","name":"gtvmdf-14","timestamp":"2021-03-24T20:50:59.402951206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ByGa8IEaJecc2xys3DlWnyS","ver":1},"status":{"status":"write"}}},{"id":"1_1616619059.407165_111.1","section":"bucket","name":"gtvmdf-14","timestamp":"2021-03-24T20:50:59.407165083Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ByGa8IEaJecc2xys3DlWnyS","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619072.516314_112.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:51:12.516314795Z","data":{"read_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":1},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":2},"status":{"status":"write"}}},{"id":"1_1616619072.523480_113.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:51:12.523480675Z","data":{"read_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":1},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619085.699597_114.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:51:25.699597319Z","data":{"read_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":2},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":3},"status":{"status":"write"}}},{"id":"1_1616619085.706880_115.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:51:25.706880905Z","data":{"read_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":2},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":3},"status":{"status":"complete"}}},{"id":"1_1616619098.934086_116.1","section":"bucket.instance","name":"gtvmdf-15:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.19","timestamp":"2021-03-24T20:51:38.934086328Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":1},"status":{"status":"write"}}},{"id":"1_1616619098.939240_117.1","section":"bucket.instance","name":"gtvmdf-15:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.19","timestamp":"2021-03-24T20:51:38.939240913Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619098.941108_118.1","section":"bucket","name":"gtvmdf-15","timestamp":"2021-03-24T20:51:38.941108427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PNuc2zTOz6GyFXbZw-fIBWb","ver":1},"status":{"status":"write"}}},{"id":"1_1616619098.945494_119.1","section":"bucket","name":"gtvmdf-15","timestamp":"2021-03-24T20:51:38.945494713Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PNuc2zTOz6GyFXbZw-fIBWb","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619098.961611_120.1","section":"bucket.instance","name":"gtvmdf-16:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.20","timestamp":"2021-03-24T20:51:38.961611938Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":1},"status":{"status":"write"}}},{"id":"1_1616619098.966402_121.1","section":"bucket.instance","name":"gtvmdf-16:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.20","timestamp":"2021-03-24T20:51:38.966402125Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619098.968020_122.1","section":"bucket","name":"gtvmdf-16","timestamp":"2021-03-24T20:51:38.968020620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_31UMZdp1WoWzvotn7Nsm__Y","ver":1},"status":{"status":"write"}}},{"id":"1_1616619098.972216_123.1","section":"bucket","name":"gtvmdf-16","timestamp":"2021-03-24T20:51:38.972216159Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_31UMZdp1WoWzvotn7Nsm__Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619098.996018_124.1","section":"bucket.instance","name":"gtvmdf-15:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.19","timestamp":"2021-03-24T20:51:38.996018519Z","data":{"read_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":1},"write_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":2},"status":{"status":"write"}}},{"id":"1_1616619099.001750_125.1","section":"bucket.instance","name":"gtvmdf-15:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.19","timestamp":"2021-03-24T20:51:39.001750396Z","data":{"read_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":1},"write_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619099.006889_126.1","section":"bucket.instance","name":"gtvmdf-16:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.20","timestamp":"2021-03-24T20:51:39.006889838Z","data":{"read_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":1},"write_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":2},"status":{"status":"write"}}},{"id":"1_1616619099.012600_127.1","section":"bucket.instance","name":"gtvmdf-16:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.20","timestamp":"2021-03-24T20:51:39.012600803Z","data":{"read_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":1},"write_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619114.027971_128.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.027971254Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":1},"status":{"status":"write"}}},{"id":"1_1616619114.032794_129.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.032794703Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619114.034888_130.1","section":"bucket","name":"gtvmdf-17","timestamp":"2021-03-24T20:51:54.034888486Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R3kKWlMS5jbJmCin0m0U6-V","ver":1},"status":{"status":"write"}}},{"id":"1_1616619114.039578_131.1","section":"bucket","name":"gtvmdf-17","timestamp":"2021-03-24T20:51:54.039578229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R3kKWlMS5jbJmCin0m0U6-V","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619114.057862_132.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.057862668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":1},"status":{"status":"write"}}},{"id":"1_1616619114.062774_133.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.062774722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619114.064381_134.1","section":"bucket","name":"gtvmdf-18","timestamp":"2021-03-24T20:51:54.064381248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pY0JjBJHuqzFvxk29Qv5iEk","ver":1},"status":{"status":"write"}}},{"id":"1_1616619114.068519_135.1","section":"bucket","name":"gtvmdf-18","timestamp":"2021-03-24T20:51:54.068519964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pY0JjBJHuqzFvxk29Qv5iEk","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619114.091214_136.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.091214680Z","data":{"read_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":1},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":2},"status":{"status":"write"}}},{"id":"1_1616619114.097005_137.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.097005230Z","data":{"read_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":1},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619114.101388_138.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.101388916Z","data":{"read_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":2},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":3},"status":{"status":"write"}}},{"id":"1_1616619114.107752_139.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.107752477Z","data":{"read_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":2},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":3},"status":{"status":"complete"}}},{"id":"1_1616619114.112716_140.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.112716205Z","data":{"read_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":1},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":2},"status":{"status":"write"}}},{"id":"1_1616619114.118374_141.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.118374960Z","data":{"read_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":1},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619114.127787_142.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.127787742Z","data":{"read_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":2},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":3},"status":{"status":"write"}}},{"id":"1_1616619114.134086_143.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.134086520Z","data":{"read_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":2},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":3},"status":{"status":"complete"}}},{"id":"1_1616619128.926611_144.1","section":"bucket.instance","name":"gtvmdf-19:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.23","timestamp":"2021-03-24T20:52:08.926611189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":1},"status":{"status":"write"}}},{"id":"1_1616619128.931253_145.1","section":"bucket.instance","name":"gtvmdf-19:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.23","timestamp":"2021-03-24T20:52:08.931253162Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619128.932758_146.1","section":"bucket","name":"gtvmdf-19","timestamp":"2021-03-24T20:52:08.932758889Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Je5wveQUSZ4tEH7MFOjdkdc","ver":1},"status":{"status":"write"}}},{"id":"1_1616619128.936408_147.1","section":"bucket","name":"gtvmdf-19","timestamp":"2021-03-24T20:52:08.936408958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Je5wveQUSZ4tEH7MFOjdkdc","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619128.951841_148.1","section":"bucket.instance","name":"gtvmdf-20:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.24","timestamp":"2021-03-24T20:52:08.951841872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":1},"status":{"status":"write"}}},{"id":"1_1616619128.956923_149.1","section":"bucket.instance","name":"gtvmdf-20:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.24","timestamp":"2021-03-24T20:52:08.956923847Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619128.958767_150.1","section":"bucket","name":"gtvmdf-20","timestamp":"2021-03-24T20:52:08.958767229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8N5Cfahn8cjfgL9uoeNSadH","ver":1},"status":{"status":"write"}}},{"id":"1_1616619128.963465_151.1","section":"bucket","name":"gtvmdf-20","timestamp":"2021-03-24T20:52:08.963465691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8N5Cfahn8cjfgL9uoeNSadH","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619128.986202_152.1","section":"bucket.instance","name":"gtvmdf-19:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.23","timestamp":"2021-03-24T20:52:08.986202010Z","data":{"read_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":1},"write_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":2},"status":{"status":"write"}}},{"id":"1_1616619128.992016_153.1","section":"bucket.instance","name":"gtvmdf-19:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.23","timestamp":"2021-03-24T20:52:08.992016781Z","data":{"read_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":1},"write_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619129.046345_154.1","section":"bucket.instance","name":"gtvmdf-20:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.24","timestamp":"2021-03-24T20:52:09.046345165Z","data":{"read_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":1},"write_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":2},"status":{"status":"write"}}},{"id":"1_1616619129.054539_155.1","section":"bucket.instance","name":"gtvmdf-20:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.24","timestamp":"2021-03-24T20:52:09.054539077Z","data":{"read_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":1},"write_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619129.123945_156.1","section":"bucket.instance","name":"gtvmdf-21:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.25","timestamp":"2021-03-24T20:52:09.123945194Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.128767_157.1","section":"bucket.instance","name":"gtvmdf-21:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.25","timestamp":"2021-03-24T20:52:09.128767956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.130274_158.1","section":"bucket","name":"gtvmdf-21","timestamp":"2021-03-24T20:52:09.130274136Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8UZ17cnIdx1ZJ8Cp87nvsOz","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.133740_159.1","section":"bucket","name":"gtvmdf-21","timestamp":"2021-03-24T20:52:09.133740102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8UZ17cnIdx1ZJ8Cp87nvsOz","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.149337_160.1","section":"bucket.instance","name":"gtvmdf-22:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.26","timestamp":"2021-03-24T20:52:09.149337081Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.154507_161.1","section":"bucket.instance","name":"gtvmdf-22:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.26","timestamp":"2021-03-24T20:52:09.154507107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.156397_162.1","section":"bucket","name":"gtvmdf-22","timestamp":"2021-03-24T20:52:09.156397349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__6dVONzzpn69bgtpbFZsSAa","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.161099_163.1","section":"bucket","name":"gtvmdf-22","timestamp":"2021-03-24T20:52:09.161099316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__6dVONzzpn69bgtpbFZsSAa","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.185839_164.1","section":"bucket.instance","name":"gtvmdf-21:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.25","timestamp":"2021-03-24T20:52:09.185839337Z","data":{"read_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":1},"write_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":2},"status":{"status":"write"}}},{"id":"1_1616619129.192122_165.1","section":"bucket.instance","name":"gtvmdf-21:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.25","timestamp":"2021-03-24T20:52:09.192122581Z","data":{"read_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":1},"write_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619129.242797_166.1","section":"bucket.instance","name":"gtvmdf-22:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.26","timestamp":"2021-03-24T20:52:09.242797437Z","data":{"read_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":1},"write_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":2},"status":{"status":"write"}}},{"id":"1_1616619129.250113_167.1","section":"bucket.instance","name":"gtvmdf-22:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.26","timestamp":"2021-03-24T20:52:09.250113393Z","data":{"read_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":1},"write_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619129.315702_168.1","section":"bucket.instance","name":"gtvmdf-23:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.27","timestamp":"2021-03-24T20:52:09.315702943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.320009_169.1","section":"bucket.instance","name":"gtvmdf-23:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.27","timestamp":"2021-03-24T20:52:09.320009574Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.321588_170.1","section":"bucket","name":"gtvmdf-23","timestamp":"2021-03-24T20:52:09.321588220Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1TrqWqWCLiV25-ESjq7E271","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.325364_171.1","section":"bucket","name":"gtvmdf-23","timestamp":"2021-03-24T20:52:09.325364400Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1TrqWqWCLiV25-ESjq7E271","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.340715_172.1","section":"bucket.instance","name":"gtvmdf-24:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.28","timestamp":"2021-03-24T20:52:09.340715588Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.345437_173.1","section":"bucket.instance","name":"gtvmdf-24:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.28","timestamp":"2021-03-24T20:52:09.345437875Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.347357_174.1","section":"bucket","name":"gtvmdf-24","timestamp":"2021-03-24T20:52:09.347357319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6FJJvGEakJfW6YWu5fy1V6x","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.351703_175.1","section":"bucket","name":"gtvmdf-24","timestamp":"2021-03-24T20:52:09.351703588Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6FJJvGEakJfW6YWu5fy1V6x","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.377737_176.1","section":"bucket.instance","name":"gtvmdf-23:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.27","timestamp":"2021-03-24T20:52:09.377737881Z","data":{"read_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":1},"write_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":2},"status":{"status":"write"}}},{"id":"1_1616619129.383378_177.1","section":"bucket.instance","name":"gtvmdf-23:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.27","timestamp":"2021-03-24T20:52:09.383378032Z","data":{"read_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":1},"write_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619130.433454_178.1","section":"bucket.instance","name":"gtvmdf-24:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.28","timestamp":"2021-03-24T20:52:10.433454675Z","data":{"read_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":1},"write_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":2},"status":{"status":"write"}}},{"id":"1_1616619130.441232_179.1","section":"bucket.instance","name":"gtvmdf-24:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.28","timestamp":"2021-03-24T20:52:10.441232968Z","data":{"read_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":1},"write_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619130.506282_180.1","section":"bucket.instance","name":"gtvmdf-25:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.29","timestamp":"2021-03-24T20:52:10.506282155Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fXX3g3kQ16Gph9Q2cFs-vHR","ver":1},"status":{"status":"write"}}},{"id":"1_1616619130.510800_181.1","section":"bucket.instance","name":"gtvmdf-25:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.29","timestamp":"2021-03-24T20:52:10.510800522Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fXX3g3kQ16Gph9Q2cFs-vHR","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619130.512288_182.1","section":"bucket","name":"gtvmdf-25","timestamp":"2021-03-24T20:52:10.512288206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hEyZngLUiA6ZyGw-3jS1LKQ","ver":1},"status":{"status":"write"}}},{"id":"1_1616619130.516322_183.1","section":"bucket","name":"gtvmdf-25","timestamp":"2021-03-24T20:52:10.516322307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hEyZngLUiA6ZyGw-3jS1LKQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619130.532853_184.1","section":"bucket.instance","name":"gtvmdf-26:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.30","timestamp":"2021-03-24T20:52:10.532853024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tS4SKRvZjZho99JH5a6148C","ver":1},"status":{"status":"write"}}},{"id":"1_1616619130.538378_185.1","section":"bucket.instance","name":"gtvmdf-26:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.30","timestamp":"2021-03-24T20:52:10.538378369Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tS4SKRvZjZho99JH5a6148C","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619130.539793_186.1","section":"bucket","name":"gtvmdf-26","timestamp":"2021-03-24T20:52:10.539793096Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZxL9Rfkew_YALii_N7IEBqX","ver":1},"status":{"status":"write"}}},{"id":"1_1616619130.543652_187.1","section":"bucket","name":"gtvmdf-26","timestamp":"2021-03-24T20:52:10.543652869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZxL9Rfkew_YALii_N7IEBqX","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619136.909575_188.1","section":"bucket.instance","name":"gtvmdf-27:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.31","timestamp":"2021-03-24T20:52:16.909575843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_d0c6LC8vx4WUhi8kjAeTuGR","ver":1},"status":{"status":"write"}}},{"id":"1_1616619136.913840_189.1","section":"bucket.instance","name":"gtvmdf-27:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.31","timestamp":"2021-03-24T20:52:16.913840638Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_d0c6LC8vx4WUhi8kjAeTuGR","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619136.915687_190.1","section":"bucket","name":"gtvmdf-27","timestamp":"2021-03-24T20:52:16.915687135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Macs6T62O19upzCNiUpxIIS","ver":1},"status":{"status":"write"}}},{"id":"1_1616619136.920010_191.1","section":"bucket","name":"gtvmdf-27","timestamp":"2021-03-24T20:52:16.920010356Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Macs6T62O19upzCNiUpxIIS","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619136.935694_192.1","section":"bucket.instance","name":"gtvmdf-28:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.32","timestamp":"2021-03-24T20:52:16.935694455Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PPoDHP_LH5U_Pp0M6wWsnd7","ver":1},"status":{"status":"write"}}},{"id":"1_1616619136.940603_193.1","section":"bucket.instance","name":"gtvmdf-28:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.32","timestamp":"2021-03-24T20:52:16.940603762Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PPoDHP_LH5U_Pp0M6wWsnd7","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619136.942415_194.1","section":"bucket","name":"gtvmdf-28","timestamp":"2021-03-24T20:52:16.942415111Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H0XNwOTOzmyG2T2RQga4hg2","ver":1},"status":{"status":"write"}}},{"id":"1_1616619136.946453_195.1","section":"bucket","name":"gtvmdf-28","timestamp":"2021-03-24T20:52:16.946453255Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H0XNwOTOzmyG2T2RQga4hg2","ver":1},"status":{"status":"complete"}}}] 2021-03-24T20:54:34.338 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'ded1abf8-1b45-4199-a116-504fb1354378', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:34.338 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'ded1abf8-1b45-4199-a116-504fb1354378', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:34.339 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period ded1abf8-1b45-4199-a116-504fb1354378 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:34.410 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:34.411 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:34.601 INFO:teuthology.orchestra.run.smithi063.stdout:[{"id":"1_1616618820.145235_26.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-24T20:47:00.145235727Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X6UefQMQBMk6T-KdROwDSmH","ver":1},"status":{"status":"write"}}},{"id":"1_1616618820.151371_27.1","section":"user","name":"rgw-multisite-test-user","timestamp":"2021-03-24T20:47:00.151371959Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_X6UefQMQBMk6T-KdROwDSmH","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618940.525007_30.1","section":"bucket.instance","name":"gtvmdf-1:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.1","timestamp":"2021-03-24T20:49:00.525007206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tgjzzXByXYIhUertDxJ3MjG","ver":1},"status":{"status":"write"}}},{"id":"1_1616618940.540912_31.1","section":"bucket.instance","name":"gtvmdf-1:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.1","timestamp":"2021-03-24T20:49:00.540912897Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tgjzzXByXYIhUertDxJ3MjG","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618940.542501_32.1","section":"bucket","name":"gtvmdf-1","timestamp":"2021-03-24T20:49:00.542501378Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xZ_E0OM_nU3o_x6hJ1NBmwF","ver":1},"status":{"status":"write"}}},{"id":"1_1616618940.546034_33.1","section":"bucket","name":"gtvmdf-1","timestamp":"2021-03-24T20:49:00.546034737Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_xZ_E0OM_nU3o_x6hJ1NBmwF","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618940.562912_34.1","section":"bucket.instance","name":"gtvmdf-2:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.2","timestamp":"2021-03-24T20:49:00.562912585Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r-IHYx4xdxs8MH2QannkeLk","ver":1},"status":{"status":"write"}}},{"id":"1_1616618940.567183_35.1","section":"bucket.instance","name":"gtvmdf-2:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.2","timestamp":"2021-03-24T20:49:00.567183856Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_r-IHYx4xdxs8MH2QannkeLk","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618940.568689_36.1","section":"bucket","name":"gtvmdf-2","timestamp":"2021-03-24T20:49:00.568689460Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jxLjNlFghqKuqp9MSw-Cwtb","ver":1},"status":{"status":"write"}}},{"id":"1_1616618940.572891_37.1","section":"bucket","name":"gtvmdf-2","timestamp":"2021-03-24T20:49:00.572891337Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jxLjNlFghqKuqp9MSw-Cwtb","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618950.394960_38.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.3","timestamp":"2021-03-24T20:49:10.394960181Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WNWA9V0cZEGDVBG0fH9bWv4","ver":1},"status":{"status":"write"}}},{"id":"1_1616618950.400191_39.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.3","timestamp":"2021-03-24T20:49:10.400191984Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_WNWA9V0cZEGDVBG0fH9bWv4","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618950.402028_40.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:10.402028094Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LKSVbZjQiFBgTZb5X0XuzWc","ver":1},"status":{"status":"write"}}},{"id":"1_1616618950.406129_41.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:10.406129774Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_LKSVbZjQiFBgTZb5X0XuzWc","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618950.421692_42.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.4","timestamp":"2021-03-24T20:49:10.421692289Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ib9mDs69W5PPcxzE8BrWHAf","ver":1},"status":{"status":"write"}}},{"id":"1_1616618950.426290_43.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.4","timestamp":"2021-03-24T20:49:10.426290203Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ib9mDs69W5PPcxzE8BrWHAf","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618950.428035_44.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:10.428035108Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbhFBHo_oGklTk1VOtTLrbX","ver":1},"status":{"status":"write"}}},{"id":"1_1616618950.432415_45.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:10.432415906Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_VbhFBHo_oGklTk1VOtTLrbX","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618956.535316_46.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.5","timestamp":"2021-03-24T20:49:16.535316144Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cjMnKx2EFjhHoXEtD9Ao2Z3","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.541236_47.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.5","timestamp":"2021-03-24T20:49:16.541236512Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cjMnKx2EFjhHoXEtD9Ao2Z3","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618956.542678_48.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:16.542678387Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yESViFgr9Rbub7DYZK_HyfE","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.545516_49.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:16.545516233Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_yESViFgr9Rbub7DYZK_HyfE","ver":1},"status":{"status":"abort"}}},{"id":"1_1616618956.565500_50.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.6","timestamp":"2021-03-24T20:49:16.565500905Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2bUZj_Da_6fOB3pOlD5P2y9","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.570055_51.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.6","timestamp":"2021-03-24T20:49:16.570055399Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2bUZj_Da_6fOB3pOlD5P2y9","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618956.571963_52.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:16.571963256Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3fzyRfY3Cujq3-aYLKy_nKM","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.575616_53.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:16.575616943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_3fzyRfY3Cujq3-aYLKy_nKM","ver":1},"status":{"status":"abort"}}},{"id":"1_1616618956.598421_54.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.7","timestamp":"2021-03-24T20:49:16.598421500Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ybNuFDA4bFl1DSj1nk48-Ml","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.603481_55.1","section":"bucket.instance","name":"gtvmdf-3:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.7","timestamp":"2021-03-24T20:49:16.603481230Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ybNuFDA4bFl1DSj1nk48-Ml","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618956.605148_56.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:16.605148162Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Obi7HCPLocdgyHzF-nOiBpd","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.608201_57.1","section":"bucket","name":"gtvmdf-3","timestamp":"2021-03-24T20:49:16.608201871Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Obi7HCPLocdgyHzF-nOiBpd","ver":1},"status":{"status":"abort"}}},{"id":"1_1616618956.649240_58.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.8","timestamp":"2021-03-24T20:49:16.649240071Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pbPLxcJ1jzhAS9F523ewOAS","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.654691_59.1","section":"bucket.instance","name":"gtvmdf-4:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.8","timestamp":"2021-03-24T20:49:16.654691122Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pbPLxcJ1jzhAS9F523ewOAS","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618956.656556_60.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:16.656556425Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kmWCjULi2Urt4nuHP9Fgda6","ver":1},"status":{"status":"write"}}},{"id":"1_1616618956.659390_61.1","section":"bucket","name":"gtvmdf-4","timestamp":"2021-03-24T20:49:16.659390794Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kmWCjULi2Urt4nuHP9Fgda6","ver":1},"status":{"status":"abort"}}},{"id":"1_1616618962.840164_62.1","section":"bucket.instance","name":"gtvmdf-5:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.9","timestamp":"2021-03-24T20:49:22.840164167Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vDfxx2c7LZmmfV2SYZvv8dT","ver":1},"status":{"status":"write"}}},{"id":"1_1616618962.846281_63.1","section":"bucket.instance","name":"gtvmdf-5:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.9","timestamp":"2021-03-24T20:49:22.846281903Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_vDfxx2c7LZmmfV2SYZvv8dT","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618962.848251_64.1","section":"bucket","name":"gtvmdf-5","timestamp":"2021-03-24T20:49:22.848251345Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":1},"status":{"status":"write"}}},{"id":"1_1616618962.852229_65.1","section":"bucket","name":"gtvmdf-5","timestamp":"2021-03-24T20:49:22.852229351Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618962.868458_66.1","section":"bucket.instance","name":"gtvmdf-6:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.10","timestamp":"2021-03-24T20:49:22.868458363Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Eu4nsv2VvM3EbZWZoASFgNS","ver":1},"status":{"status":"write"}}},{"id":"1_1616618962.873247_67.1","section":"bucket.instance","name":"gtvmdf-6:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.10","timestamp":"2021-03-24T20:49:22.873247875Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Eu4nsv2VvM3EbZWZoASFgNS","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618962.874924_68.1","section":"bucket","name":"gtvmdf-6","timestamp":"2021-03-24T20:49:22.874924380Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":1},"status":{"status":"write"}}},{"id":"1_1616618962.879387_69.1","section":"bucket","name":"gtvmdf-6","timestamp":"2021-03-24T20:49:22.879387934Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618969.082901_70.1","section":"bucket","name":"gtvmdf-5","timestamp":"2021-03-24T20:49:29.082901809Z","data":{"read_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":1},"write_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":2},"status":{"status":"remove"}}},{"id":"1_1616618969.088732_71.1","section":"bucket","name":"gtvmdf-5","timestamp":"2021-03-24T20:49:29.088732072Z","data":{"read_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":1},"write_version":{"tag":"_zZK58zo3DO94BAMvVLvKjQC","ver":2},"status":{"status":"complete"}}},{"id":"1_1616618969.124397_72.1","section":"bucket","name":"gtvmdf-6","timestamp":"2021-03-24T20:49:29.124397201Z","data":{"read_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":1},"write_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":2},"status":{"status":"remove"}}},{"id":"1_1616618969.129804_73.1","section":"bucket","name":"gtvmdf-6","timestamp":"2021-03-24T20:49:29.129804567Z","data":{"read_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":1},"write_version":{"tag":"_btC6WVSM1PgcNhRzzW9B0Ft","ver":2},"status":{"status":"complete"}}},{"id":"1_1616618975.259328_74.1","section":"bucket.instance","name":"gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11","timestamp":"2021-03-24T20:49:35.259328006Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sxZH7m-QFlmDpKA8AgtzhPF","ver":1},"status":{"status":"write"}}},{"id":"1_1616618975.264835_75.1","section":"bucket.instance","name":"gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11","timestamp":"2021-03-24T20:49:35.264835737Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sxZH7m-QFlmDpKA8AgtzhPF","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618975.266208_76.1","section":"bucket","name":"gtvmdf-7","timestamp":"2021-03-24T20:49:35.266208098Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zz5yVnTJtCVXtuxON_a8ehY","ver":1},"status":{"status":"write"}}},{"id":"1_1616618975.270002_77.1","section":"bucket","name":"gtvmdf-7","timestamp":"2021-03-24T20:49:35.270002818Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Zz5yVnTJtCVXtuxON_a8ehY","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618975.284878_78.1","section":"bucket.instance","name":"gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12","timestamp":"2021-03-24T20:49:35.284878435Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KMZLUo4l0O9JD4Hr04kuPbL","ver":1},"status":{"status":"write"}}},{"id":"1_1616618975.289559_79.1","section":"bucket.instance","name":"gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12","timestamp":"2021-03-24T20:49:35.289559510Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KMZLUo4l0O9JD4Hr04kuPbL","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618975.291378_80.1","section":"bucket","name":"gtvmdf-8","timestamp":"2021-03-24T20:49:35.291378616Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jy6J99ZlUqP_RnQt-0vqafJ","ver":1},"status":{"status":"write"}}},{"id":"1_1616618975.295785_81.1","section":"bucket","name":"gtvmdf-8","timestamp":"2021-03-24T20:49:35.295785394Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_jy6J99ZlUqP_RnQt-0vqafJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618986.091302_82.1","section":"bucket.instance","name":"gtvmdf-9:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.13","timestamp":"2021-03-24T20:49:46.091302407Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-Vo7HKGnWkmFy0AVuOrx8vD","ver":1},"status":{"status":"write"}}},{"id":"1_1616618986.095790_83.1","section":"bucket.instance","name":"gtvmdf-9:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.13","timestamp":"2021-03-24T20:49:46.095790175Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_-Vo7HKGnWkmFy0AVuOrx8vD","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618986.097257_84.1","section":"bucket","name":"gtvmdf-9","timestamp":"2021-03-24T20:49:46.097257755Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oylq7sQHahJX4um-hkauus5","ver":1},"status":{"status":"write"}}},{"id":"1_1616618986.101275_85.1","section":"bucket","name":"gtvmdf-9","timestamp":"2021-03-24T20:49:46.101275951Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_oylq7sQHahJX4um-hkauus5","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618986.116223_86.1","section":"bucket.instance","name":"gtvmdf-10:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.14","timestamp":"2021-03-24T20:49:46.116223321Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lPn06j2Mk7rBKrns9S4gsnh","ver":1},"status":{"status":"write"}}},{"id":"1_1616618986.121070_87.1","section":"bucket.instance","name":"gtvmdf-10:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.14","timestamp":"2021-03-24T20:49:46.121070252Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_lPn06j2Mk7rBKrns9S4gsnh","ver":1},"status":{"status":"complete"}}},{"id":"1_1616618986.123005_88.1","section":"bucket","name":"gtvmdf-10","timestamp":"2021-03-24T20:49:46.123005877Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9l4AizNcZyFC7-zC0CB1TeJ","ver":1},"status":{"status":"write"}}},{"id":"1_1616618986.127919_89.1","section":"bucket","name":"gtvmdf-10","timestamp":"2021-03-24T20:49:46.127919319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_9l4AizNcZyFC7-zC0CB1TeJ","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619007.041460_90.1","section":"bucket.instance","name":"gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15","timestamp":"2021-03-24T20:50:07.041460956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":1},"status":{"status":"write"}}},{"id":"1_1616619007.047026_91.1","section":"bucket.instance","name":"gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15","timestamp":"2021-03-24T20:50:07.047026910Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619007.048478_92.1","section":"bucket","name":"gtvmdf-11","timestamp":"2021-03-24T20:50:07.048478376Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AgLAtS2oyLPaHP6unNPtmVC","ver":1},"status":{"status":"write"}}},{"id":"1_1616619007.052429_93.1","section":"bucket","name":"gtvmdf-11","timestamp":"2021-03-24T20:50:07.052429307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_AgLAtS2oyLPaHP6unNPtmVC","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619007.067245_94.1","section":"bucket.instance","name":"gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16","timestamp":"2021-03-24T20:50:07.067245681Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":1},"status":{"status":"write"}}},{"id":"1_1616619007.072112_95.1","section":"bucket.instance","name":"gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16","timestamp":"2021-03-24T20:50:07.072112238Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619007.074206_96.1","section":"bucket","name":"gtvmdf-12","timestamp":"2021-03-24T20:50:07.074206526Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DQRLU6jS7zHVvjx0sgi1qPV","ver":1},"status":{"status":"write"}}},{"id":"1_1616619007.079276_97.1","section":"bucket","name":"gtvmdf-12","timestamp":"2021-03-24T20:50:07.079276614Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_DQRLU6jS7zHVvjx0sgi1qPV","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619007.103931_98.1","section":"bucket.instance","name":"gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15","timestamp":"2021-03-24T20:50:07.103931395Z","data":{"read_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":1},"write_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":2},"status":{"status":"write"}}},{"id":"1_1616619007.110653_99.1","section":"bucket.instance","name":"gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15","timestamp":"2021-03-24T20:50:07.110653160Z","data":{"read_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":1},"write_version":{"tag":"_Y4hdW7qVQZK0s6sycC2tu4a","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619007.116106_100.1","section":"bucket.instance","name":"gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16","timestamp":"2021-03-24T20:50:07.116106696Z","data":{"read_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":1},"write_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":2},"status":{"status":"write"}}},{"id":"1_1616619007.123420_101.1","section":"bucket.instance","name":"gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16","timestamp":"2021-03-24T20:50:07.123420937Z","data":{"read_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":1},"write_version":{"tag":"_v4rji85Yc_lPu_QoPmA_a5b","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619035.929342_102.1","section":"bucket.instance","name":"gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17","timestamp":"2021-03-24T20:50:35.929342176Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":1},"status":{"status":"write"}}},{"id":"1_1616619035.934528_103.1","section":"bucket.instance","name":"gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17","timestamp":"2021-03-24T20:50:35.934528828Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619035.936547_104.1","section":"bucket","name":"gtvmdf-13","timestamp":"2021-03-24T20:50:35.936547600Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTs-jZYrEVVtkrfK3uz3f0b","ver":1},"status":{"status":"write"}}},{"id":"1_1616619035.940636_105.1","section":"bucket","name":"gtvmdf-13","timestamp":"2021-03-24T20:50:35.940636057Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kTs-jZYrEVVtkrfK3uz3f0b","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619035.945564_106.1","section":"bucket.instance","name":"gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17","timestamp":"2021-03-24T20:50:35.945564609Z","data":{"read_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":1},"write_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":2},"status":{"status":"write"}}},{"id":"1_1616619035.950831_107.1","section":"bucket.instance","name":"gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17","timestamp":"2021-03-24T20:50:35.950831213Z","data":{"read_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":1},"write_version":{"tag":"_hq3eSqhWEejELkjkVJSwLGc","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619059.395270_108.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:50:59.395270616Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":1},"status":{"status":"write"}}},{"id":"1_1616619059.401521_109.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:50:59.401521462Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619059.402951_110.1","section":"bucket","name":"gtvmdf-14","timestamp":"2021-03-24T20:50:59.402951206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ByGa8IEaJecc2xys3DlWnyS","ver":1},"status":{"status":"write"}}},{"id":"1_1616619059.407165_111.1","section":"bucket","name":"gtvmdf-14","timestamp":"2021-03-24T20:50:59.407165083Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ByGa8IEaJecc2xys3DlWnyS","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619072.516314_112.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:51:12.516314795Z","data":{"read_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":1},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":2},"status":{"status":"write"}}},{"id":"1_1616619072.523480_113.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:51:12.523480675Z","data":{"read_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":1},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619085.699597_114.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:51:25.699597319Z","data":{"read_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":2},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":3},"status":{"status":"write"}}},{"id":"1_1616619085.706880_115.1","section":"bucket.instance","name":"gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18","timestamp":"2021-03-24T20:51:25.706880905Z","data":{"read_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":2},"write_version":{"tag":"_TGwJc4NEZOIycy1eCqsBA46","ver":3},"status":{"status":"complete"}}},{"id":"1_1616619098.934086_116.1","section":"bucket.instance","name":"gtvmdf-15:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.19","timestamp":"2021-03-24T20:51:38.934086328Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":1},"status":{"status":"write"}}},{"id":"1_1616619098.939240_117.1","section":"bucket.instance","name":"gtvmdf-15:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.19","timestamp":"2021-03-24T20:51:38.939240913Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619098.941108_118.1","section":"bucket","name":"gtvmdf-15","timestamp":"2021-03-24T20:51:38.941108427Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PNuc2zTOz6GyFXbZw-fIBWb","ver":1},"status":{"status":"write"}}},{"id":"1_1616619098.945494_119.1","section":"bucket","name":"gtvmdf-15","timestamp":"2021-03-24T20:51:38.945494713Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PNuc2zTOz6GyFXbZw-fIBWb","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619098.961611_120.1","section":"bucket.instance","name":"gtvmdf-16:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.20","timestamp":"2021-03-24T20:51:38.961611938Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":1},"status":{"status":"write"}}},{"id":"1_1616619098.966402_121.1","section":"bucket.instance","name":"gtvmdf-16:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.20","timestamp":"2021-03-24T20:51:38.966402125Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619098.968020_122.1","section":"bucket","name":"gtvmdf-16","timestamp":"2021-03-24T20:51:38.968020620Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_31UMZdp1WoWzvotn7Nsm__Y","ver":1},"status":{"status":"write"}}},{"id":"1_1616619098.972216_123.1","section":"bucket","name":"gtvmdf-16","timestamp":"2021-03-24T20:51:38.972216159Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_31UMZdp1WoWzvotn7Nsm__Y","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619098.996018_124.1","section":"bucket.instance","name":"gtvmdf-15:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.19","timestamp":"2021-03-24T20:51:38.996018519Z","data":{"read_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":1},"write_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":2},"status":{"status":"write"}}},{"id":"1_1616619099.001750_125.1","section":"bucket.instance","name":"gtvmdf-15:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.19","timestamp":"2021-03-24T20:51:39.001750396Z","data":{"read_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":1},"write_version":{"tag":"_Q867xkuezB4j9Pgpx5BMjUe","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619099.006889_126.1","section":"bucket.instance","name":"gtvmdf-16:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.20","timestamp":"2021-03-24T20:51:39.006889838Z","data":{"read_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":1},"write_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":2},"status":{"status":"write"}}},{"id":"1_1616619099.012600_127.1","section":"bucket.instance","name":"gtvmdf-16:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.20","timestamp":"2021-03-24T20:51:39.012600803Z","data":{"read_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":1},"write_version":{"tag":"_NKkbsZUChy6npDqisiccBTM","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619114.027971_128.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.027971254Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":1},"status":{"status":"write"}}},{"id":"1_1616619114.032794_129.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.032794703Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619114.034888_130.1","section":"bucket","name":"gtvmdf-17","timestamp":"2021-03-24T20:51:54.034888486Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R3kKWlMS5jbJmCin0m0U6-V","ver":1},"status":{"status":"write"}}},{"id":"1_1616619114.039578_131.1","section":"bucket","name":"gtvmdf-17","timestamp":"2021-03-24T20:51:54.039578229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_R3kKWlMS5jbJmCin0m0U6-V","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619114.057862_132.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.057862668Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":1},"status":{"status":"write"}}},{"id":"1_1616619114.062774_133.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.062774722Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619114.064381_134.1","section":"bucket","name":"gtvmdf-18","timestamp":"2021-03-24T20:51:54.064381248Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pY0JjBJHuqzFvxk29Qv5iEk","ver":1},"status":{"status":"write"}}},{"id":"1_1616619114.068519_135.1","section":"bucket","name":"gtvmdf-18","timestamp":"2021-03-24T20:51:54.068519964Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_pY0JjBJHuqzFvxk29Qv5iEk","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619114.091214_136.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.091214680Z","data":{"read_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":1},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":2},"status":{"status":"write"}}},{"id":"1_1616619114.097005_137.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.097005230Z","data":{"read_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":1},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619114.101388_138.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.101388916Z","data":{"read_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":2},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":3},"status":{"status":"write"}}},{"id":"1_1616619114.107752_139.1","section":"bucket.instance","name":"gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21","timestamp":"2021-03-24T20:51:54.107752477Z","data":{"read_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":2},"write_version":{"tag":"_cf6ZpkN3-HypVWuVcWN39Ic","ver":3},"status":{"status":"complete"}}},{"id":"1_1616619114.112716_140.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.112716205Z","data":{"read_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":1},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":2},"status":{"status":"write"}}},{"id":"1_1616619114.118374_141.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.118374960Z","data":{"read_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":1},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619114.127787_142.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.127787742Z","data":{"read_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":2},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":3},"status":{"status":"write"}}},{"id":"1_1616619114.134086_143.1","section":"bucket.instance","name":"gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22","timestamp":"2021-03-24T20:51:54.134086520Z","data":{"read_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":2},"write_version":{"tag":"_wn6QQMRU7-cIOkIlC6QKDyz","ver":3},"status":{"status":"complete"}}},{"id":"1_1616619128.926611_144.1","section":"bucket.instance","name":"gtvmdf-19:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.23","timestamp":"2021-03-24T20:52:08.926611189Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":1},"status":{"status":"write"}}},{"id":"1_1616619128.931253_145.1","section":"bucket.instance","name":"gtvmdf-19:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.23","timestamp":"2021-03-24T20:52:08.931253162Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619128.932758_146.1","section":"bucket","name":"gtvmdf-19","timestamp":"2021-03-24T20:52:08.932758889Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Je5wveQUSZ4tEH7MFOjdkdc","ver":1},"status":{"status":"write"}}},{"id":"1_1616619128.936408_147.1","section":"bucket","name":"gtvmdf-19","timestamp":"2021-03-24T20:52:08.936408958Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Je5wveQUSZ4tEH7MFOjdkdc","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619128.951841_148.1","section":"bucket.instance","name":"gtvmdf-20:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.24","timestamp":"2021-03-24T20:52:08.951841872Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":1},"status":{"status":"write"}}},{"id":"1_1616619128.956923_149.1","section":"bucket.instance","name":"gtvmdf-20:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.24","timestamp":"2021-03-24T20:52:08.956923847Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619128.958767_150.1","section":"bucket","name":"gtvmdf-20","timestamp":"2021-03-24T20:52:08.958767229Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8N5Cfahn8cjfgL9uoeNSadH","ver":1},"status":{"status":"write"}}},{"id":"1_1616619128.963465_151.1","section":"bucket","name":"gtvmdf-20","timestamp":"2021-03-24T20:52:08.963465691Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8N5Cfahn8cjfgL9uoeNSadH","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619128.986202_152.1","section":"bucket.instance","name":"gtvmdf-19:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.23","timestamp":"2021-03-24T20:52:08.986202010Z","data":{"read_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":1},"write_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":2},"status":{"status":"write"}}},{"id":"1_1616619128.992016_153.1","section":"bucket.instance","name":"gtvmdf-19:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.23","timestamp":"2021-03-24T20:52:08.992016781Z","data":{"read_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":1},"write_version":{"tag":"_s2r5zT_fmUsKBajOKh1H_8o","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619129.046345_154.1","section":"bucket.instance","name":"gtvmdf-20:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.24","timestamp":"2021-03-24T20:52:09.046345165Z","data":{"read_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":1},"write_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":2},"status":{"status":"write"}}},{"id":"1_1616619129.054539_155.1","section":"bucket.instance","name":"gtvmdf-20:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.24","timestamp":"2021-03-24T20:52:09.054539077Z","data":{"read_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":1},"write_version":{"tag":"_KjOhQwQyjjCe9PWUKcgGJ6_","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619129.123945_156.1","section":"bucket.instance","name":"gtvmdf-21:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.25","timestamp":"2021-03-24T20:52:09.123945194Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.128767_157.1","section":"bucket.instance","name":"gtvmdf-21:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.25","timestamp":"2021-03-24T20:52:09.128767956Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.130274_158.1","section":"bucket","name":"gtvmdf-21","timestamp":"2021-03-24T20:52:09.130274136Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8UZ17cnIdx1ZJ8Cp87nvsOz","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.133740_159.1","section":"bucket","name":"gtvmdf-21","timestamp":"2021-03-24T20:52:09.133740102Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8UZ17cnIdx1ZJ8Cp87nvsOz","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.149337_160.1","section":"bucket.instance","name":"gtvmdf-22:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.26","timestamp":"2021-03-24T20:52:09.149337081Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.154507_161.1","section":"bucket.instance","name":"gtvmdf-22:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.26","timestamp":"2021-03-24T20:52:09.154507107Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.156397_162.1","section":"bucket","name":"gtvmdf-22","timestamp":"2021-03-24T20:52:09.156397349Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__6dVONzzpn69bgtpbFZsSAa","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.161099_163.1","section":"bucket","name":"gtvmdf-22","timestamp":"2021-03-24T20:52:09.161099316Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"__6dVONzzpn69bgtpbFZsSAa","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.185839_164.1","section":"bucket.instance","name":"gtvmdf-21:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.25","timestamp":"2021-03-24T20:52:09.185839337Z","data":{"read_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":1},"write_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":2},"status":{"status":"write"}}},{"id":"1_1616619129.192122_165.1","section":"bucket.instance","name":"gtvmdf-21:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.25","timestamp":"2021-03-24T20:52:09.192122581Z","data":{"read_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":1},"write_version":{"tag":"_winL8LCuJY4tvjvrt5iYa8I","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619129.242797_166.1","section":"bucket.instance","name":"gtvmdf-22:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.26","timestamp":"2021-03-24T20:52:09.242797437Z","data":{"read_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":1},"write_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":2},"status":{"status":"write"}}},{"id":"1_1616619129.250113_167.1","section":"bucket.instance","name":"gtvmdf-22:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.26","timestamp":"2021-03-24T20:52:09.250113393Z","data":{"read_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":1},"write_version":{"tag":"_zQvxAkY1Izn7x2obKxPDN3d","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619129.315702_168.1","section":"bucket.instance","name":"gtvmdf-23:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.27","timestamp":"2021-03-24T20:52:09.315702943Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.320009_169.1","section":"bucket.instance","name":"gtvmdf-23:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.27","timestamp":"2021-03-24T20:52:09.320009574Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.321588_170.1","section":"bucket","name":"gtvmdf-23","timestamp":"2021-03-24T20:52:09.321588220Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1TrqWqWCLiV25-ESjq7E271","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.325364_171.1","section":"bucket","name":"gtvmdf-23","timestamp":"2021-03-24T20:52:09.325364400Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_1TrqWqWCLiV25-ESjq7E271","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.340715_172.1","section":"bucket.instance","name":"gtvmdf-24:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.28","timestamp":"2021-03-24T20:52:09.340715588Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.345437_173.1","section":"bucket.instance","name":"gtvmdf-24:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.28","timestamp":"2021-03-24T20:52:09.345437875Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.347357_174.1","section":"bucket","name":"gtvmdf-24","timestamp":"2021-03-24T20:52:09.347357319Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6FJJvGEakJfW6YWu5fy1V6x","ver":1},"status":{"status":"write"}}},{"id":"1_1616619129.351703_175.1","section":"bucket","name":"gtvmdf-24","timestamp":"2021-03-24T20:52:09.351703588Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_6FJJvGEakJfW6YWu5fy1V6x","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619129.377737_176.1","section":"bucket.instance","name":"gtvmdf-23:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.27","timestamp":"2021-03-24T20:52:09.377737881Z","data":{"read_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":1},"write_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":2},"status":{"status":"write"}}},{"id":"1_1616619129.383378_177.1","section":"bucket.instance","name":"gtvmdf-23:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.27","timestamp":"2021-03-24T20:52:09.383378032Z","data":{"read_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":1},"write_version":{"tag":"_UHchMYybAy1iGIt8cfI6lzN","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619130.433454_178.1","section":"bucket.instance","name":"gtvmdf-24:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.28","timestamp":"2021-03-24T20:52:10.433454675Z","data":{"read_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":1},"write_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":2},"status":{"status":"write"}}},{"id":"1_1616619130.441232_179.1","section":"bucket.instance","name":"gtvmdf-24:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.28","timestamp":"2021-03-24T20:52:10.441232968Z","data":{"read_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":1},"write_version":{"tag":"_2fO80d_2003VoCPhfKgelQU","ver":2},"status":{"status":"complete"}}},{"id":"1_1616619130.506282_180.1","section":"bucket.instance","name":"gtvmdf-25:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.29","timestamp":"2021-03-24T20:52:10.506282155Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fXX3g3kQ16Gph9Q2cFs-vHR","ver":1},"status":{"status":"write"}}},{"id":"1_1616619130.510800_181.1","section":"bucket.instance","name":"gtvmdf-25:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.29","timestamp":"2021-03-24T20:52:10.510800522Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_fXX3g3kQ16Gph9Q2cFs-vHR","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619130.512288_182.1","section":"bucket","name":"gtvmdf-25","timestamp":"2021-03-24T20:52:10.512288206Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hEyZngLUiA6ZyGw-3jS1LKQ","ver":1},"status":{"status":"write"}}},{"id":"1_1616619130.516322_183.1","section":"bucket","name":"gtvmdf-25","timestamp":"2021-03-24T20:52:10.516322307Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_hEyZngLUiA6ZyGw-3jS1LKQ","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619130.532853_184.1","section":"bucket.instance","name":"gtvmdf-26:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.30","timestamp":"2021-03-24T20:52:10.532853024Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tS4SKRvZjZho99JH5a6148C","ver":1},"status":{"status":"write"}}},{"id":"1_1616619130.538378_185.1","section":"bucket.instance","name":"gtvmdf-26:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.30","timestamp":"2021-03-24T20:52:10.538378369Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_tS4SKRvZjZho99JH5a6148C","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619130.539793_186.1","section":"bucket","name":"gtvmdf-26","timestamp":"2021-03-24T20:52:10.539793096Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZxL9Rfkew_YALii_N7IEBqX","ver":1},"status":{"status":"write"}}},{"id":"1_1616619130.543652_187.1","section":"bucket","name":"gtvmdf-26","timestamp":"2021-03-24T20:52:10.543652869Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_ZxL9Rfkew_YALii_N7IEBqX","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619136.909575_188.1","section":"bucket.instance","name":"gtvmdf-27:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.31","timestamp":"2021-03-24T20:52:16.909575843Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_d0c6LC8vx4WUhi8kjAeTuGR","ver":1},"status":{"status":"write"}}},{"id":"1_1616619136.913840_189.1","section":"bucket.instance","name":"gtvmdf-27:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.31","timestamp":"2021-03-24T20:52:16.913840638Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_d0c6LC8vx4WUhi8kjAeTuGR","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619136.915687_190.1","section":"bucket","name":"gtvmdf-27","timestamp":"2021-03-24T20:52:16.915687135Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Macs6T62O19upzCNiUpxIIS","ver":1},"status":{"status":"write"}}},{"id":"1_1616619136.920010_191.1","section":"bucket","name":"gtvmdf-27","timestamp":"2021-03-24T20:52:16.920010356Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_Macs6T62O19upzCNiUpxIIS","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619136.935694_192.1","section":"bucket.instance","name":"gtvmdf-28:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.32","timestamp":"2021-03-24T20:52:16.935694455Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PPoDHP_LH5U_Pp0M6wWsnd7","ver":1},"status":{"status":"write"}}},{"id":"1_1616619136.940603_193.1","section":"bucket.instance","name":"gtvmdf-28:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.32","timestamp":"2021-03-24T20:52:16.940603762Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_PPoDHP_LH5U_Pp0M6wWsnd7","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619136.942415_194.1","section":"bucket","name":"gtvmdf-28","timestamp":"2021-03-24T20:52:16.942415111Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H0XNwOTOzmyG2T2RQga4hg2","ver":1},"status":{"status":"write"}}},{"id":"1_1616619136.946453_195.1","section":"bucket","name":"gtvmdf-28","timestamp":"2021-03-24T20:52:16.946453255Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_H0XNwOTOzmyG2T2RQga4hg2","ver":1},"status":{"status":"complete"}}}] 2021-03-24T20:54:34.603 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '9c8fef3d-af5e-4308-a22f-c09517bb5569', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:34.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '9c8fef3d-af5e-4308-a22f-c09517bb5569', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:34.604 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 9c8fef3d-af5e-4308-a22f-c09517bb5569 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:34.644 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:34.644 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:34.834 INFO:teuthology.orchestra.run.smithi052.stdout:[{"id":"1_1616619204.309833_62.1","section":"bucket.instance","name":"gtvmdf-29:b3573a14-e2bf-4e98-b96a-79727832ba59.5156.1","timestamp":"2021-03-24T20:53:24.309833330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YnG2IrDmhZhmnQP-lbRvm_R","ver":1},"status":{"status":"write"}}},{"id":"1_1616619204.314077_63.1","section":"bucket.instance","name":"gtvmdf-29:b3573a14-e2bf-4e98-b96a-79727832ba59.5156.1","timestamp":"2021-03-24T20:53:24.314077796Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YnG2IrDmhZhmnQP-lbRvm_R","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619204.315589_64.1","section":"bucket","name":"gtvmdf-29","timestamp":"2021-03-24T20:53:24.315589031Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sszANkH13j4QhXmZHMI0Dhu","ver":1},"status":{"status":"write"}}},{"id":"1_1616619204.320382_65.1","section":"bucket","name":"gtvmdf-29","timestamp":"2021-03-24T20:53:24.320382528Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sszANkH13j4QhXmZHMI0Dhu","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619204.357371_66.1","section":"bucket.instance","name":"gtvmdf-30:b3573a14-e2bf-4e98-b96a-79727832ba59.5156.2","timestamp":"2021-03-24T20:53:24.357371211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l41drln-gWCs4aD8gFEmQRN","ver":1},"status":{"status":"write"}}},{"id":"1_1616619204.361943_67.1","section":"bucket.instance","name":"gtvmdf-30:b3573a14-e2bf-4e98-b96a-79727832ba59.5156.2","timestamp":"2021-03-24T20:53:24.361943056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l41drln-gWCs4aD8gFEmQRN","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619204.363472_68.1","section":"bucket","name":"gtvmdf-30","timestamp":"2021-03-24T20:53:24.363472314Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CQNQWVvijcWvKD4MU60yklj","ver":1},"status":{"status":"write"}}},{"id":"1_1616619204.367468_69.1","section":"bucket","name":"gtvmdf-30","timestamp":"2021-03-24T20:53:24.367468526Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CQNQWVvijcWvKD4MU60yklj","ver":1},"status":{"status":"complete"}}}] 2021-03-24T20:54:34.835 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '9c8fef3d-af5e-4308-a22f-c09517bb5569', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:34.836 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '9c8fef3d-af5e-4308-a22f-c09517bb5569', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:34.836 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 9c8fef3d-af5e-4308-a22f-c09517bb5569 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:34.875 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:34.875 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:35.062 INFO:teuthology.orchestra.run.smithi063.stdout:[{"id":"1_1616619204.309833_62.1","section":"bucket.instance","name":"gtvmdf-29:b3573a14-e2bf-4e98-b96a-79727832ba59.5156.1","timestamp":"2021-03-24T20:53:24.309833330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YnG2IrDmhZhmnQP-lbRvm_R","ver":1},"status":{"status":"write"}}},{"id":"1_1616619204.314077_63.1","section":"bucket.instance","name":"gtvmdf-29:b3573a14-e2bf-4e98-b96a-79727832ba59.5156.1","timestamp":"2021-03-24T20:53:24.314077796Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YnG2IrDmhZhmnQP-lbRvm_R","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619204.315589_64.1","section":"bucket","name":"gtvmdf-29","timestamp":"2021-03-24T20:53:24.315589031Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sszANkH13j4QhXmZHMI0Dhu","ver":1},"status":{"status":"write"}}},{"id":"1_1616619204.320382_65.1","section":"bucket","name":"gtvmdf-29","timestamp":"2021-03-24T20:53:24.320382528Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sszANkH13j4QhXmZHMI0Dhu","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619204.357371_66.1","section":"bucket.instance","name":"gtvmdf-30:b3573a14-e2bf-4e98-b96a-79727832ba59.5156.2","timestamp":"2021-03-24T20:53:24.357371211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l41drln-gWCs4aD8gFEmQRN","ver":1},"status":{"status":"write"}}},{"id":"1_1616619204.361943_67.1","section":"bucket.instance","name":"gtvmdf-30:b3573a14-e2bf-4e98-b96a-79727832ba59.5156.2","timestamp":"2021-03-24T20:53:24.361943056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l41drln-gWCs4aD8gFEmQRN","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619204.363472_68.1","section":"bucket","name":"gtvmdf-30","timestamp":"2021-03-24T20:53:24.363472314Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CQNQWVvijcWvKD4MU60yklj","ver":1},"status":{"status":"write"}}},{"id":"1_1616619204.367468_69.1","section":"bucket","name":"gtvmdf-30","timestamp":"2021-03-24T20:53:24.367468526Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CQNQWVvijcWvKD4MU60yklj","ver":1},"status":{"status":"complete"}}}] 2021-03-24T20:54:35.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '9c8fef3d-af5e-4308-a22f-c09517bb5569', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:35.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '9c8fef3d-af5e-4308-a22f-c09517bb5569', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:35.064 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 9c8fef3d-af5e-4308-a22f-c09517bb5569 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:35.140 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:35.141 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:35.330 INFO:teuthology.orchestra.run.smithi063.stdout:[{"id":"1_1616619204.309833_62.1","section":"bucket.instance","name":"gtvmdf-29:b3573a14-e2bf-4e98-b96a-79727832ba59.5156.1","timestamp":"2021-03-24T20:53:24.309833330Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YnG2IrDmhZhmnQP-lbRvm_R","ver":1},"status":{"status":"write"}}},{"id":"1_1616619204.314077_63.1","section":"bucket.instance","name":"gtvmdf-29:b3573a14-e2bf-4e98-b96a-79727832ba59.5156.1","timestamp":"2021-03-24T20:53:24.314077796Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_YnG2IrDmhZhmnQP-lbRvm_R","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619204.315589_64.1","section":"bucket","name":"gtvmdf-29","timestamp":"2021-03-24T20:53:24.315589031Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sszANkH13j4QhXmZHMI0Dhu","ver":1},"status":{"status":"write"}}},{"id":"1_1616619204.320382_65.1","section":"bucket","name":"gtvmdf-29","timestamp":"2021-03-24T20:53:24.320382528Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sszANkH13j4QhXmZHMI0Dhu","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619204.357371_66.1","section":"bucket.instance","name":"gtvmdf-30:b3573a14-e2bf-4e98-b96a-79727832ba59.5156.2","timestamp":"2021-03-24T20:53:24.357371211Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l41drln-gWCs4aD8gFEmQRN","ver":1},"status":{"status":"write"}}},{"id":"1_1616619204.361943_67.1","section":"bucket.instance","name":"gtvmdf-30:b3573a14-e2bf-4e98-b96a-79727832ba59.5156.2","timestamp":"2021-03-24T20:53:24.361943056Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_l41drln-gWCs4aD8gFEmQRN","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619204.363472_68.1","section":"bucket","name":"gtvmdf-30","timestamp":"2021-03-24T20:53:24.363472314Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CQNQWVvijcWvKD4MU60yklj","ver":1},"status":{"status":"write"}}},{"id":"1_1616619204.367468_69.1","section":"bucket","name":"gtvmdf-30","timestamp":"2021-03-24T20:53:24.367468526Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_CQNQWVvijcWvKD4MU60yklj","ver":1},"status":{"status":"complete"}}}] 2021-03-24T20:54:35.330 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'c083ad76-6371-41ed-840a-4187dd1c26c6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:35.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'c083ad76-6371-41ed-840a-4187dd1c26c6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:35.332 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period c083ad76-6371-41ed-840a-4187dd1c26c6 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:35.370 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:35.371 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:35.559 INFO:teuthology.orchestra.run.smithi052.stdout:[{"id":"1_1616619270.518232_216.1","section":"bucket.instance","name":"gtvmdf-31:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.1","timestamp":"2021-03-24T20:54:30.518232603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCwQaRvOedsc7xOkztbmRgt","ver":1},"status":{"status":"write"}}},{"id":"1_1616619270.523484_217.1","section":"bucket.instance","name":"gtvmdf-31:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.1","timestamp":"2021-03-24T20:54:30.523484392Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCwQaRvOedsc7xOkztbmRgt","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619270.525163_218.1","section":"bucket","name":"gtvmdf-31","timestamp":"2021-03-24T20:54:30.525163065Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8VnBjH9nZfAFaIdxXS5CnU2","ver":1},"status":{"status":"write"}}},{"id":"1_1616619270.529047_219.1","section":"bucket","name":"gtvmdf-31","timestamp":"2021-03-24T20:54:30.529047304Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8VnBjH9nZfAFaIdxXS5CnU2","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619270.553630_220.1","section":"bucket.instance","name":"gtvmdf-32:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.2","timestamp":"2021-03-24T20:54:30.553630696Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sDexxrS_PPTSbeIBT2r4AD5","ver":1},"status":{"status":"write"}}},{"id":"1_1616619270.558689_221.1","section":"bucket.instance","name":"gtvmdf-32:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.2","timestamp":"2021-03-24T20:54:30.558689719Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sDexxrS_PPTSbeIBT2r4AD5","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619270.560434_222.1","section":"bucket","name":"gtvmdf-32","timestamp":"2021-03-24T20:54:30.560434264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KB4PMR-cZk7i28XfVYxg2hy","ver":1},"status":{"status":"write"}}},{"id":"1_1616619270.564805_223.1","section":"bucket","name":"gtvmdf-32","timestamp":"2021-03-24T20:54:30.564805982Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KB4PMR-cZk7i28XfVYxg2hy","ver":1},"status":{"status":"complete"}}}] 2021-03-24T20:54:35.559 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'c083ad76-6371-41ed-840a-4187dd1c26c6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:35.560 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'c083ad76-6371-41ed-840a-4187dd1c26c6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:35.561 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period c083ad76-6371-41ed-840a-4187dd1c26c6 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:35.602 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:35.602 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:35.803 INFO:teuthology.orchestra.run.smithi063.stdout:[{"id":"1_1616619270.518232_216.1","section":"bucket.instance","name":"gtvmdf-31:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.1","timestamp":"2021-03-24T20:54:30.518232603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCwQaRvOedsc7xOkztbmRgt","ver":1},"status":{"status":"write"}}},{"id":"1_1616619270.523484_217.1","section":"bucket.instance","name":"gtvmdf-31:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.1","timestamp":"2021-03-24T20:54:30.523484392Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCwQaRvOedsc7xOkztbmRgt","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619270.525163_218.1","section":"bucket","name":"gtvmdf-31","timestamp":"2021-03-24T20:54:30.525163065Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8VnBjH9nZfAFaIdxXS5CnU2","ver":1},"status":{"status":"write"}}},{"id":"1_1616619270.529047_219.1","section":"bucket","name":"gtvmdf-31","timestamp":"2021-03-24T20:54:30.529047304Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8VnBjH9nZfAFaIdxXS5CnU2","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619270.553630_220.1","section":"bucket.instance","name":"gtvmdf-32:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.2","timestamp":"2021-03-24T20:54:30.553630696Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sDexxrS_PPTSbeIBT2r4AD5","ver":1},"status":{"status":"write"}}},{"id":"1_1616619270.558689_221.1","section":"bucket.instance","name":"gtvmdf-32:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.2","timestamp":"2021-03-24T20:54:30.558689719Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sDexxrS_PPTSbeIBT2r4AD5","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619270.560434_222.1","section":"bucket","name":"gtvmdf-32","timestamp":"2021-03-24T20:54:30.560434264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KB4PMR-cZk7i28XfVYxg2hy","ver":1},"status":{"status":"write"}}},{"id":"1_1616619270.564805_223.1","section":"bucket","name":"gtvmdf-32","timestamp":"2021-03-24T20:54:30.564805982Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KB4PMR-cZk7i28XfVYxg2hy","ver":1},"status":{"status":"complete"}}}] 2021-03-24T20:54:35.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'c083ad76-6371-41ed-840a-4187dd1c26c6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:35.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'c083ad76-6371-41ed-840a-4187dd1c26c6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:35.804 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period c083ad76-6371-41ed-840a-4187dd1c26c6 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:35.883 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:35.884 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:36.088 INFO:teuthology.orchestra.run.smithi063.stdout:[{"id":"1_1616619270.518232_216.1","section":"bucket.instance","name":"gtvmdf-31:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.1","timestamp":"2021-03-24T20:54:30.518232603Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCwQaRvOedsc7xOkztbmRgt","ver":1},"status":{"status":"write"}}},{"id":"1_1616619270.523484_217.1","section":"bucket.instance","name":"gtvmdf-31:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.1","timestamp":"2021-03-24T20:54:30.523484392Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_kCwQaRvOedsc7xOkztbmRgt","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619270.525163_218.1","section":"bucket","name":"gtvmdf-31","timestamp":"2021-03-24T20:54:30.525163065Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8VnBjH9nZfAFaIdxXS5CnU2","ver":1},"status":{"status":"write"}}},{"id":"1_1616619270.529047_219.1","section":"bucket","name":"gtvmdf-31","timestamp":"2021-03-24T20:54:30.529047304Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_8VnBjH9nZfAFaIdxXS5CnU2","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619270.553630_220.1","section":"bucket.instance","name":"gtvmdf-32:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.2","timestamp":"2021-03-24T20:54:30.553630696Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sDexxrS_PPTSbeIBT2r4AD5","ver":1},"status":{"status":"write"}}},{"id":"1_1616619270.558689_221.1","section":"bucket.instance","name":"gtvmdf-32:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.2","timestamp":"2021-03-24T20:54:30.558689719Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_sDexxrS_PPTSbeIBT2r4AD5","ver":1},"status":{"status":"complete"}}},{"id":"1_1616619270.560434_222.1","section":"bucket","name":"gtvmdf-32","timestamp":"2021-03-24T20:54:30.560434264Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KB4PMR-cZk7i28XfVYxg2hy","ver":1},"status":{"status":"write"}}},{"id":"1_1616619270.564805_223.1","section":"bucket","name":"gtvmdf-32","timestamp":"2021-03-24T20:54:30.564805982Z","data":{"read_version":{"tag":"","ver":0},"write_version":{"tag":"_KB4PMR-cZk7i28XfVYxg2hy","ver":1},"status":{"status":"complete"}}}] 2021-03-24T20:54:36.089 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:54:36.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:54:36.090 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T20:54:36.126 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:36.127 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:36.362 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:54:36.363 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:54:36.363 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:54:36.400 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:36.401 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:36.728 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:54:36.730 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:54:36.731 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:54:36.806 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:36.806 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:37.099 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'ded1abf8-1b45-4199-a116-504fb1354378', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:37.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'ded1abf8-1b45-4199-a116-504fb1354378', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:37.100 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period ded1abf8-1b45-4199-a116-504fb1354378 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:37.138 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:37.139 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:37.334 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2021-03-24T20:54:37.334 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'ded1abf8-1b45-4199-a116-504fb1354378', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:37.335 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'ded1abf8-1b45-4199-a116-504fb1354378', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:37.335 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period ded1abf8-1b45-4199-a116-504fb1354378 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:37.375 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:37.376 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:37.562 INFO:teuthology.orchestra.run.smithi063.stdout:[] 2021-03-24T20:54:37.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'ded1abf8-1b45-4199-a116-504fb1354378', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:37.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'ded1abf8-1b45-4199-a116-504fb1354378', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:37.563 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period ded1abf8-1b45-4199-a116-504fb1354378 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:37.647 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:37.648 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:37.847 INFO:teuthology.orchestra.run.smithi063.stdout:[] 2021-03-24T20:54:37.847 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', '9c8fef3d-af5e-4308-a22f-c09517bb5569', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:37.848 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', '9c8fef3d-af5e-4308-a22f-c09517bb5569', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:37.848 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period 9c8fef3d-af5e-4308-a22f-c09517bb5569 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:37.887 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:37.887 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:38.081 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2021-03-24T20:54:38.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '9c8fef3d-af5e-4308-a22f-c09517bb5569', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:38.082 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '9c8fef3d-af5e-4308-a22f-c09517bb5569', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:38.083 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 9c8fef3d-af5e-4308-a22f-c09517bb5569 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:38.123 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:38.123 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:38.307 INFO:teuthology.orchestra.run.smithi063.stdout:[] 2021-03-24T20:54:38.307 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', '9c8fef3d-af5e-4308-a22f-c09517bb5569', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:38.308 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', '9c8fef3d-af5e-4308-a22f-c09517bb5569', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:38.308 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period 9c8fef3d-af5e-4308-a22f-c09517bb5569 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:38.387 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:38.387 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:38.584 INFO:teuthology.orchestra.run.smithi063.stdout:[] 2021-03-24T20:54:38.584 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'list', '--period', 'c083ad76-6371-41ed-840a-4187dd1c26c6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:38.585 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'list', '--period', 'c083ad76-6371-41ed-840a-4187dd1c26c6', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:38.585 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog list --period c083ad76-6371-41ed-840a-4187dd1c26c6 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:38.625 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:38.625 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:38.823 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2021-03-24T20:54:38.824 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'c083ad76-6371-41ed-840a-4187dd1c26c6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:38.824 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'c083ad76-6371-41ed-840a-4187dd1c26c6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:38.825 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period c083ad76-6371-41ed-840a-4187dd1c26c6 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:38.863 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:38.863 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:39.062 INFO:teuthology.orchestra.run.smithi063.stdout:[] 2021-03-24T20:54:39.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['mdlog', 'list', '--period', 'c083ad76-6371-41ed-840a-4187dd1c26c6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:39.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'mdlog', 'list', '--period', 'c083ad76-6371-41ed-840a-4187dd1c26c6', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:39.063 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 mdlog list --period c083ad76-6371-41ed-840a-4187dd1c26c6 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:39.145 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:39.145 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:39.328 INFO:teuthology.orchestra.run.smithi063.stdout:[] 2021-03-24T20:54:39.329 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_period_incremental_sync ... ok 2021-03-24T20:54:39.332 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-33 2021-03-24T20:54:39.358 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-34 2021-03-24T20:54:39.503 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:39.504 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:39.505 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:39.547 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:39.548 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:39.723 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616619137.144173_40.1","last_update":"2021-03-24T20:52:17.144173Z"},{"marker":"1_1616619030.611627_106.2","last_update":"2021-03-24T20:50:30.611627Z"},{"marker":"1_1616619279.418138_120.1","last_update":"2021-03-24T20:54:39.418138Z"},{"marker":"1_1616619052.619482_77.1","last_update":"2021-03-24T20:50:52.619482Z"}] 2021-03-24T20:54:39.723 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-24T20:54:39.724 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:39.724 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:39.725 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:39.768 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:39.768 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:39.961 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619140.633549_99.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:20.633549Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619052.619482_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:52.619482Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T20:54:39.962 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=2 master=1_1616619279.418138_120.1 target=1_1616619140.633549_99.2 2021-03-24T20:54:40.477 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T20:54:40.472+0000 7f2d34ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T20:54:40.478 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T20:54:40.472+0000 7f442d7f2700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T20:54:40.479 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T20:54:40.476+0000 7f2d21ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T20:54:40.480 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T20:54:40.476+0000 7f44297ea700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T20:54:44.964 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:44.965 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:44.966 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:45.011 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:45.204 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619052.619482_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:52.619482Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T20:54:45.205 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-24T20:54:45.206 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:45.206 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:45.207 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:45.250 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:45.250 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:45.437 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616619137.144173_40.1","last_update":"2021-03-24T20:52:17.144173Z"},{"marker":"1_1616619030.611627_106.2","last_update":"2021-03-24T20:50:30.611627Z"},{"marker":"1_1616619280.644430_130.1","last_update":"2021-03-24T20:54:40.644430Z"},{"marker":"1_1616619052.619482_77.1","last_update":"2021-03-24T20:50:52.619482Z"}] 2021-03-24T20:54:45.438 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-24T20:54:45.439 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:45.439 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:45.440 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:45.482 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:45.483 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:45.747 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14224098923836610154},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619052.619482_77.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:52.619482Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T20:54:45.747 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-24T20:54:45.748 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:45.748 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:45.749 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:45.829 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:45.830 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:46.020 INFO:teuthology.orchestra.run.smithi063.stdout:[{"marker":"1_1616619137.927569_51.1","last_update":"2021-03-24T20:52:17.927569Z"},{"marker":"1_1616619031.564526_105.2","last_update":"2021-03-24T20:50:31.564526Z"},{"marker":"1_1616619280.659935_163.1","last_update":"2021-03-24T20:54:40.659935Z"},{"marker":"1_1616619053.571247_88.1","last_update":"2021-03-24T20:50:53.571247Z"}] 2021-03-24T20:54:46.021 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-24T20:54:46.021 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:46.022 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:46.023 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:46.066 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:46.067 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:46.251 INFO:teuthology.orchestra.run.smithi052.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14589021674366829938},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.927569_51.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.927569Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619031.564526_105.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:31.564526Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.659935_163.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.659935Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619053.571247_88.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:53.571247Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T20:54:46.252 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-24T20:54:46.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:54:46.254 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'autotrim', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:54:46.254 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog autotrim --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T20:54:46.334 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:46.335 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:46.519 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:46.520 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'list', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:46.520 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog list --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:46.599 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:46.600 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:46.786 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2021-03-24T20:54:46.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:54:46.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'autotrim', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:54:46.788 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog autotrim --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:54:46.832 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:46.832 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:47.038 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:47.038 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'list', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:54:47.039 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog list --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:54:47.122 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:47.123 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:47.317 INFO:teuthology.orchestra.run.smithi063.stdout:[] 2021-03-24T20:54:47.318 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_datalog_autotrim ... ok 2021-03-24T20:54:47.321 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:54:47.321 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:54:47.322 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=false --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:54:47.402 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:47.403 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:47.428 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:54:47.424+0000 7f926a63b080 0 failed reading obj info from .rgw.root:zone_info.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df: (2) No such file or directory 2021-03-24T20:54:47.429 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:54:47.424+0000 7f926a63b080 0 WARNING: could not read zone params for zone id=662a64c9-f3f4-4dbc-9258-dc3b70a5b4df name=test-zone1 2021-03-24T20:54:47.442 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"HFYQ9V2T1L1XP8R6","secret_key":"EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8"} 2021-03-24T20:54:47.442 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:54:47.443 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:54:47.444 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:54:47.525 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:54:47.526 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:54:47.729 INFO:teuthology.orchestra.run.smithi063.stderr:Sending period to new master zone 662a64c9-f3f4-4dbc-9258-dc3b70a5b4df 2021-03-24T20:54:48.818 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","epoch":2,"predecessor_uuid":"9c8fef3d-af5e-4308-a22f-c09517bb5569","sync_status":["","","","1_1616619204.367468_69.1"],"period_map":{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","zones":[{"id":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","name":"test-zone1","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"91064457-7aa0-4413-b153-d6cf440dd094","name":"test-zone3","endpoints":["http://smithi063.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","endpoints":["http://smithi063.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","val":2285303741},{"key":"91064457-7aa0-4413-b153-d6cf440dd094","val":575160010},{"key":"b3573a14-e2bf-4e98-b96a-79727832ba59","val":746531257}]},"master_zonegroup":"test-zonegroup","master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","realm_name":"test-realm","realm_epoch":4} 2021-03-24T20:54:48.819 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to false 2021-03-24T20:55:48.820 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-35 2021-03-24T20:55:48.903 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:55:48.903 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:55:48.904 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:55:48.943 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:55:48.944 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:55:49.101 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T20:55:49.130 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616619348.856719_254.1","last_update":"2021-03-24T20:55:48.856719Z"}] 2021-03-24T20:55:49.131 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:55:49.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:55:49.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:55:49.133 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:55:49.172 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:55:49.172 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:55:49.372 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619279.384103_234.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:39.384103Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:55:49.372 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616619348.856719_254.1 target=1_1616619279.384103_234.1 2021-03-24T20:55:54.374 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:55:54.375 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:55:54.376 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:55:54.417 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:55:54.418 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:55:54.617 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619279.384103_234.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:39.384103Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:55:54.618 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616619348.856719_254.1 target=1_1616619279.384103_234.1 2021-03-24T20:55:59.620 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:55:59.622 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:55:59.623 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:55:59.665 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:55:59.665 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:55:59.869 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619279.384103_234.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:39.384103Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:55:59.869 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616619348.856719_254.1 target=1_1616619279.384103_234.1 2021-03-24T20:56:04.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:04.873 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:04.874 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:56:04.917 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:56:04.917 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:56:05.117 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619279.384103_234.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:39.384103Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:56:05.117 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616619348.856719_254.1 target=1_1616619279.384103_234.1 2021-03-24T20:56:10.119 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:10.121 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:10.122 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:56:10.163 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:56:10.163 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:56:10.361 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619279.384103_234.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:39.384103Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:56:10.361 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616619348.856719_254.1 target=1_1616619279.384103_234.1 2021-03-24T20:56:15.363 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:15.369 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:15.370 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:56:15.411 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:56:15.411 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:56:15.621 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619279.384103_234.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:39.384103Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:56:15.622 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616619348.856719_254.1 target=1_1616619279.384103_234.1 2021-03-24T20:56:20.623 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:20.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:20.630 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:56:20.673 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:56:20.674 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:56:20.877 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619348.856719_254.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:55:48.856719Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:56:20.878 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:56:20.878 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:56:20.879 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:20.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:20.880 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:56:20.957 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:56:20.957 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:56:21.218 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619279.384103_234.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:39.384103Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:56:21.219 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1616619348.856719_254.1 target=1_1616619279.384103_234.1 2021-03-24T20:56:26.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:26.228 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:26.228 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:56:26.270 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:56:26.271 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:56:26.528 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619279.384103_234.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:39.384103Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:56:26.529 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1616619348.856719_254.1 target=1_1616619279.384103_234.1 2021-03-24T20:56:31.531 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:31.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:31.532 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:56:31.573 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:56:31.574 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:56:31.831 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619279.384103_234.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:39.384103Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:56:31.832 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1616619348.856719_254.1 target=1_1616619279.384103_234.1 2021-03-24T20:56:36.833 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:36.834 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:36.835 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:56:36.877 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:56:36.878 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:56:37.136 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619279.384103_234.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:39.384103Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:56:37.136 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1616619348.856719_254.1 target=1_1616619279.384103_234.1 2021-03-24T20:56:42.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:42.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:42.140 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:56:42.180 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:56:42.180 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:56:42.428 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619279.384103_234.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:39.384103Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:56:42.429 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1616619348.856719_254.1 target=1_1616619279.384103_234.1 2021-03-24T20:56:47.430 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:47.431 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:47.432 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:56:47.473 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:56:47.473 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:56:47.737 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619279.384103_234.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:39.384103Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:56:47.737 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1616619348.856719_254.1 target=1_1616619279.384103_234.1 2021-03-24T20:56:49.919 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T20:56:49.918+0000 7f442fff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T20:56:52.738 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:52.739 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:56:52.740 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:56:52.780 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:56:52.780 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:56:53.037 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619348.856719_254.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:55:48.856719Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:56:53.038 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:56:53.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=662a64c9-f3f4-4dbc-9258-dc3b70a5b4df', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:56:53.066 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=662a64c9-f3f4-4dbc-9258-dc3b70a5b4df', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:56:53.067 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone=662a64c9-f3f4-4dbc-9258-dc3b70a5b4df --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:56:53.123 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:56:53.124 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:56:53.152 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:56:53.150+0000 7ff626840080 0 failed reading obj info from .rgw.root:zone_info.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df: (2) No such file or directory 2021-03-24T20:56:53.153 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:56:53.150+0000 7ff626840080 0 WARNING: could not read zone params for zone id=662a64c9-f3f4-4dbc-9258-dc3b70a5b4df name=test-zone1 2021-03-24T20:56:53.169 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"HFYQ9V2T1L1XP8R6","secret_key":"EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8"} 2021-03-24T20:56:53.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:56:53.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:56:53.171 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:56:53.252 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:56:53.253 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:56:53.451 INFO:teuthology.orchestra.run.smithi063.stderr:Sending period to new master zone 662a64c9-f3f4-4dbc-9258-dc3b70a5b4df 2021-03-24T20:56:54.542 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","epoch":3,"predecessor_uuid":"9c8fef3d-af5e-4308-a22f-c09517bb5569","sync_status":["","","","1_1616619204.367468_69.1"],"period_map":{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","zones":[{"id":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","name":"test-zone1","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"91064457-7aa0-4413-b153-d6cf440dd094","name":"test-zone3","endpoints":["http://smithi063.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","endpoints":["http://smithi063.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"false","sync_from":[],"redirect_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df"}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","val":2285303741},{"key":"91064457-7aa0-4413-b153-d6cf440dd094","val":575160010},{"key":"b3573a14-e2bf-4e98-b96a-79727832ba59","val":746531257}]},"master_zonegroup":"test-zonegroup","master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","realm_name":"test-realm","realm_epoch":4} 2021-03-24T20:56:54.543 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "662a64c9-f3f4-4dbc-9258-dc3b70a5b4df" 2021-03-24T20:57:54.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:57:54.879 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--sync-from-all=true', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:57:54.880 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --sync-from-all=true --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:57:54.921 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:57:54.921 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:57:54.949 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:57:54.946+0000 7f0b14404080 0 failed reading obj info from .rgw.root:zone_info.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df: (2) No such file or directory 2021-03-24T20:57:54.950 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:57:54.946+0000 7f0b14404080 0 WARNING: could not read zone params for zone id=662a64c9-f3f4-4dbc-9258-dc3b70a5b4df name=test-zone1 2021-03-24T20:57:54.966 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"HFYQ9V2T1L1XP8R6","secret_key":"EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8"} 2021-03-24T20:57:54.966 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:57:54.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:57:54.968 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:57:55.049 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:57:55.050 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:57:55.230 INFO:teuthology.orchestra.run.smithi063.stderr:Sending period to new master zone 662a64c9-f3f4-4dbc-9258-dc3b70a5b4df 2021-03-24T20:57:56.321 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","epoch":4,"predecessor_uuid":"9c8fef3d-af5e-4308-a22f-c09517bb5569","sync_status":["","","","1_1616619204.367468_69.1"],"period_map":{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","zones":[{"id":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","name":"test-zone1","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"91064457-7aa0-4413-b153-d6cf440dd094","name":"test-zone3","endpoints":["http://smithi063.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","endpoints":["http://smithi063.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df"}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","val":2285303741},{"key":"91064457-7aa0-4413-b153-d6cf440dd094","val":575160010},{"key":"b3573a14-e2bf-4e98-b96a-79727832ba59","val":746531257}]},"master_zonegroup":"test-zonegroup","master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","realm_name":"test-realm","realm_epoch":4} 2021-03-24T20:57:56.327 INFO:rgw_multi.tests:Set sync_from_all flag on zone test-zone2 to true 2021-03-24T20:58:56.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:58:56.331 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'modify', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--redirect-zone=', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:58:56.332 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone modify --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --redirect-zone= --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:58:56.372 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:58:56.372 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:58:56.400 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:58:56.397+0000 7f975b28e080 0 failed reading obj info from .rgw.root:zone_info.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df: (2) No such file or directory 2021-03-24T20:58:56.400 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:58:56.397+0000 7f975b28e080 0 WARNING: could not read zone params for zone id=662a64c9-f3f4-4dbc-9258-dc3b70a5b4df name=test-zone1 2021-03-24T20:58:56.413 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","domain_root":"test-zone2.rgw.meta:root","control_pool":"test-zone2.rgw.control","gc_pool":"test-zone2.rgw.log:gc","lc_pool":"test-zone2.rgw.log:lc","log_pool":"test-zone2.rgw.log","intent_log_pool":"test-zone2.rgw.log:intent","usage_log_pool":"test-zone2.rgw.log:usage","roles_pool":"test-zone2.rgw.meta:roles","reshard_pool":"test-zone2.rgw.log:reshard","user_keys_pool":"test-zone2.rgw.meta:users.keys","user_email_pool":"test-zone2.rgw.meta:users.email","user_swift_pool":"test-zone2.rgw.meta:users.swift","user_uid_pool":"test-zone2.rgw.meta:users.uid","otp_pool":"test-zone2.rgw.otp","system_key":{"access_key":"HFYQ9V2T1L1XP8R6","secret_key":"EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"test-zone2.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"test-zone2.rgw.buckets.data","compression_type":"random"}},"data_extra_pool":"test-zone2.rgw.buckets.non-ec","index_type":0}}],"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8"} 2021-03-24T20:58:56.414 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:58:56.414 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:58:56.415 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:58:56.491 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:58:56.492 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:58:56.689 INFO:teuthology.orchestra.run.smithi063.stderr:Sending period to new master zone 662a64c9-f3f4-4dbc-9258-dc3b70a5b4df 2021-03-24T20:58:57.775 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","epoch":5,"predecessor_uuid":"9c8fef3d-af5e-4308-a22f-c09517bb5569","sync_status":["","","","1_1616619204.367468_69.1"],"period_map":{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","zones":[{"id":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","name":"test-zone1","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"91064457-7aa0-4413-b153-d6cf440dd094","name":"test-zone3","endpoints":["http://smithi063.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","endpoints":["http://smithi063.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","val":2285303741},{"key":"91064457-7aa0-4413-b153-d6cf440dd094","val":575160010},{"key":"b3573a14-e2bf-4e98-b96a-79727832ba59","val":746531257}]},"master_zonegroup":"test-zonegroup","master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","realm_name":"test-realm","realm_epoch":4} 2021-03-24T20:58:57.776 INFO:rgw_multi.tests:Set redirect_zone zone test-zone2 to "" 2021-03-24T20:59:57.779 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multi_zone_redirect ... ok 2021-03-24T20:59:57.780 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:59:57.781 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:59:57.782 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:59:57.823 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:59:57.823 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:59:57.979 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T20:59:58.011 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616619348.856719_254.1","last_update":"2021-03-24T20:55:48.856719Z"}] 2021-03-24T20:59:58.011 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T20:59:58.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:59:58.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:59:58.013 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:59:58.055 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:59:58.055 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:59:58.254 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619348.856719_254.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:55:48.856719Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:59:58.255 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T20:59:58.256 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T20:59:58.256 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:59:58.257 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T20:59:58.258 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T20:59:58.338 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:59:58.339 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:59:58.600 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619348.856719_254.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:55:48.856719Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T20:59:58.601 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T20:59:58.601 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HFYQ9V2T1L1XP8R6', '--secret', 'EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:59:58.603 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--access-key', 'HFYQ9V2T1L1XP8R6', '--secret', 'EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:59:58.604 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone create --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --access-key HFYQ9V2T1L1XP8R6 --secret EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:59:58.684 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:59:58.685 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T20:59:58.741 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:59:58.736+0000 7fb86a863080 0 failed reading obj info from .rgw.root:zone_info.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df: (2) No such file or directory 2021-03-24T20:59:58.741 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T20:59:58.736+0000 7fb86a863080 0 WARNING: could not read zone params for zone id=662a64c9-f3f4-4dbc-9258-dc3b70a5b4df name=test-zone1 2021-03-24T20:59:58.755 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"8cf0b1f0-8d79-41e0-8638-df784402601d","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"HFYQ9V2T1L1XP8R6","secret_key":"EjIfafKkUUN3GD8FCy8ER2dTGm19CDMS"},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8"} 2021-03-24T20:59:58.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:59:58.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'period', 'update', '--rgw-zone', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T20:59:58.757 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 period update --rgw-zone remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T20:59:58.837 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T20:59:58.838 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:06.421 INFO:teuthology.orchestra.run.smithi063.stderr:Sending period to new master zone 662a64c9-f3f4-4dbc-9258-dc3b70a5b4df 2021-03-24T21:00:07.500 INFO:teuthology.orchestra.run.smithi063.stdout:{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","epoch":6,"predecessor_uuid":"9c8fef3d-af5e-4308-a22f-c09517bb5569","sync_status":["","","","1_1616619204.367468_69.1"],"period_map":{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","zones":[{"id":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","name":"test-zone1","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"8cf0b1f0-8d79-41e0-8638-df784402601d","name":"remove","endpoints":[],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"91064457-7aa0-4413-b153-d6cf440dd094","name":"test-zone3","endpoints":["http://smithi063.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","endpoints":["http://smithi063.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","val":2285303741},{"key":"8cf0b1f0-8d79-41e0-8638-df784402601d","val":3545897935},{"key":"91064457-7aa0-4413-b153-d6cf440dd094","val":575160010},{"key":"b3573a14-e2bf-4e98-b96a-79727832ba59","val":746531257}]},"master_zonegroup":"test-zonegroup","master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","realm_name":"test-realm","realm_epoch":4} 2021-03-24T21:00:07.501 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:07.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:07.503 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:00:07.543 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:07.543 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:07.569 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T21:00:07.567+0000 7fdcfcd9f080 0 failed reading obj info from .rgw.root:zone_info.91064457-7aa0-4413-b153-d6cf440dd094: (2) No such file or directory 2021-03-24T21:00:07.570 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T21:00:07.567+0000 7fdcfcd9f080 0 WARNING: could not read zone params for zone id=91064457-7aa0-4413-b153-d6cf440dd094 name=test-zone3 2021-03-24T21:00:07.570 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T21:00:07.567+0000 7fdcfcd9f080 0 failed reading obj info from .rgw.root:zone_info.b3573a14-e2bf-4e98-b96a-79727832ba59: (2) No such file or directory 2021-03-24T21:00:07.571 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T21:00:07.567+0000 7fdcfcd9f080 0 WARNING: could not read zone params for zone id=b3573a14-e2bf-4e98-b96a-79727832ba59 name=test-zone2 2021-03-24T21:00:07.578 INFO:teuthology.orchestra.run.smithi052.stdout:{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","zones":[{"id":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","name":"test-zone1","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"91064457-7aa0-4413-b153-d6cf440dd094","name":"test-zone3","endpoints":["http://smithi063.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","endpoints":["http://smithi063.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","sync_policy":{"groups":[]}} 2021-03-24T21:00:07.578 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:07.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'remove', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--rgw-zone', 'remove', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:07.580 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup remove --rgw-zonegroup test-zonegroup --rgw-realm test-realm --rgw-zone remove --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:00:07.660 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:07.661 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:07.683 INFO:teuthology.orchestra.run.smithi052.stderr:zone name remove not found in zonegroup test-zonegroup 2021-03-24T21:00:07.690 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-03-24T21:00:07.691 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:07.692 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'zone', 'delete', '--rgw-zone', 'remove', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:07.693 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 zone delete --rgw-zone remove --cluster c2 --debug-rgw 0 --debug-ms 0 2021-03-24T21:00:07.730 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:07.730 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:07.774 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T21:00:07.772+0000 7efd87aec080 0 failed reading obj info from .rgw.root:zone_info.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df: (2) No such file or directory 2021-03-24T21:00:07.775 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T21:00:07.772+0000 7efd87aec080 0 WARNING: could not read zone params for zone id=662a64c9-f3f4-4dbc-9258-dc3b70a5b4df name=test-zone1 2021-03-24T21:00:07.779 INFO:teuthology.orchestra.run.smithi063.stderr:2021-03-24T21:00:07.776+0000 7efd87aec080 0 zone id 8cf0b1f0-8d79-41e0-8638-df784402601d is not a part of zonegroup default 2021-03-24T21:00:07.793 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:07.794 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:07.794 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:00:07.835 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:07.836 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:07.873 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T21:00:07.871+0000 7f98e58c5080 0 WARNING: can't generate connection for zone 8cf0b1f0-8d79-41e0-8638-df784402601d id remove: no endpoints defined 2021-03-24T21:00:07.989 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:00:07.984+0000 7f42ac740700 -1 failed to decode period 2021-03-24T21:00:07.990 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:00:07.984+0000 7f2b43e77700 -1 failed to decode period 2021-03-24T21:00:08.010 INFO:teuthology.orchestra.run.smithi052.stdout:{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","epoch":7,"predecessor_uuid":"9c8fef3d-af5e-4308-a22f-c09517bb5569","sync_status":["","","","1_1616619204.367468_69.1"],"period_map":{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","zones":[{"id":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","name":"test-zone1","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"91064457-7aa0-4413-b153-d6cf440dd094","name":"test-zone3","endpoints":["http://smithi063.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","endpoints":["http://smithi063.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","val":2285303741},{"key":"91064457-7aa0-4413-b153-d6cf440dd094","val":575160010},{"key":"b3573a14-e2bf-4e98-b96a-79727832ba59","val":746531257}]},"master_zonegroup":"test-zonegroup","master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","realm_name":"test-realm","realm_epoch":4} 2021-03-24T21:00:08.011 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zonegroup_remove ... ok 2021-03-24T21:00:08.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:08.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'create', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:08.013 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup create --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:00:08.088 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:08.089 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:08.116 INFO:teuthology.orchestra.run.smithi052.stdout:{"id":"0abf8664-2d38-43ac-a9bf-c57a60251d26","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"","zones":[],"placement_targets":[],"default_placement":"","realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","sync_policy":{"groups":[]}} 2021-03-24T21:00:08.117 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:08.117 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'create', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:08.118 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone create --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:00:08.198 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:08.199 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:08.254 INFO:teuthology.orchestra.run.smithi052.stdout:{"id":"45f0b75b-c748-40d3-9e63-b7538cbb080a","name":"remove","domain_root":"remove.rgw.meta:root","control_pool":"remove.rgw.control","gc_pool":"remove.rgw.log:gc","lc_pool":"remove.rgw.log:lc","log_pool":"remove.rgw.log","intent_log_pool":"remove.rgw.log:intent","usage_log_pool":"remove.rgw.log:usage","roles_pool":"remove.rgw.meta:roles","reshard_pool":"remove.rgw.log:reshard","user_keys_pool":"remove.rgw.meta:users.keys","user_email_pool":"remove.rgw.meta:users.email","user_swift_pool":"remove.rgw.meta:users.swift","user_uid_pool":"remove.rgw.meta:users.uid","otp_pool":"remove.rgw.otp","system_key":{"access_key":"","secret_key":""},"placement_pools":[{"key":"default-placement","val":{"index_pool":"remove.rgw.buckets.index","storage_classes":{"STANDARD":{"data_pool":"remove.rgw.buckets.data"}},"data_extra_pool":"remove.rgw.buckets.non-ec","index_type":0}}],"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8"} 2021-03-24T21:00:08.255 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:08.255 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:08.256 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:00:08.332 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:08.333 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:08.523 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:00:08.520+0000 7f2b3de6b700 -1 failed to decode period 2021-03-24T21:00:08.523 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:00:08.520+0000 7f42a6f35700 -1 failed to decode period 2021-03-24T21:00:08.546 INFO:teuthology.orchestra.run.smithi052.stdout:{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","epoch":8,"predecessor_uuid":"9c8fef3d-af5e-4308-a22f-c09517bb5569","sync_status":["","","","1_1616619204.367468_69.1"],"period_map":{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","zonegroups":[{"id":"0abf8664-2d38-43ac-a9bf-c57a60251d26","name":"remove_zg","api_name":"remove_zg","is_master":"false","endpoints":[],"hostnames":[],"hostnames_s3website":[],"master_zone":"45f0b75b-c748-40d3-9e63-b7538cbb080a","zones":[{"id":"45f0b75b-c748-40d3-9e63-b7538cbb080a","name":"remove","endpoints":[],"log_meta":"false","log_data":"false","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","sync_policy":{"groups":[]}},{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","zones":[{"id":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","name":"test-zone1","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"91064457-7aa0-4413-b153-d6cf440dd094","name":"test-zone3","endpoints":["http://smithi063.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","endpoints":["http://smithi063.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"45f0b75b-c748-40d3-9e63-b7538cbb080a","val":890741903},{"key":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","val":2285303741},{"key":"91064457-7aa0-4413-b153-d6cf440dd094","val":575160010},{"key":"b3573a14-e2bf-4e98-b96a-79727832ba59","val":746531257}]},"master_zonegroup":"test-zonegroup","master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","realm_name":"test-realm","realm_epoch":4} 2021-03-24T21:00:08.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:08.548 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zone', 'delete', '--rgw-zone', 'remove', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:08.549 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zone delete --rgw-zone remove --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:00:08.628 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:08.629 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:08.665 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T21:00:08.663+0000 7f43300ec080 0 zone id 45f0b75b-c748-40d3-9e63-b7538cbb080a is not a part of zonegroup test-zonegroup 2021-03-24T21:00:08.666 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T21:00:08.663+0000 7f43300ec080 0 zone id 45f0b75b-c748-40d3-9e63-b7538cbb080a is not a part of zonegroup default 2021-03-24T21:00:08.676 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:08.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:08.678 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:00:08.758 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:08.759 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:08.797 INFO:teuthology.orchestra.run.smithi052.stderr:2021-03-24T21:00:08.794+0000 7fece80f1080 0 ERROR: zonegroup remove_zg has a non existent master zone 2021-03-24T21:00:08.803 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-03-24T21:00:08.804 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:08.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'zonegroup', 'delete', '--rgw-zonegroup', 'remove_zg', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:08.805 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 zonegroup delete --rgw-zonegroup remove_zg --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:00:08.886 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:08.887 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:08.917 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:08.918 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'period', 'update', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--commit', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:08.919 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 period update --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --commit --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:00:09.001 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:09.003 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:09.207 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:00:09.204+0000 7f2b7feef700 -1 failed to decode period 2021-03-24T21:00:09.208 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:00:09.204+0000 7f4294710700 -1 failed to decode period 2021-03-24T21:00:09.233 INFO:teuthology.orchestra.run.smithi052.stdout:{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","epoch":9,"predecessor_uuid":"9c8fef3d-af5e-4308-a22f-c09517bb5569","sync_status":["","","","1_1616619204.367468_69.1"],"period_map":{"id":"c083ad76-6371-41ed-840a-4187dd1c26c6","zonegroups":[{"id":"test-zonegroup","name":"test-zonegroup","api_name":"","is_master":"true","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"hostnames":[],"hostnames_s3website":[],"master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","zones":[{"id":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","name":"test-zone1","endpoints":["http://smithi052.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""},{"id":"91064457-7aa0-4413-b153-d6cf440dd094","name":"test-zone3","endpoints":["http://smithi063.front.sepia.ceph.com:8001"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"pubsub","sync_from_all":"false","sync_from":["test-zone1"],"redirect_zone":""},{"id":"b3573a14-e2bf-4e98-b96a-79727832ba59","name":"test-zone2","endpoints":["http://smithi063.front.sepia.ceph.com:8000"],"log_meta":"false","log_data":"true","bucket_index_max_shards":11,"read_only":"false","tier_type":"","sync_from_all":"true","sync_from":[],"redirect_zone":""}],"placement_targets":[{"name":"default-placement","tags":[],"storage_classes":["STANDARD"]}],"default_placement":"default-placement","realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","sync_policy":{"groups":[]}}],"short_zone_ids":[{"key":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","val":2285303741},{"key":"91064457-7aa0-4413-b153-d6cf440dd094","val":575160010},{"key":"b3573a14-e2bf-4e98-b96a-79727832ba59","val":746531257}]},"master_zonegroup":"test-zonegroup","master_zone":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","period_config":{"bucket_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1},"user_quota":{"enabled":false,"check_on_raw":false,"max_size":-1,"max_size_kb":0,"max_objects":-1}},"realm_id":"e90890b4-d954-49f4-b444-8f808b8250e8","realm_name":"test-realm","realm_epoch":4} 2021-03-24T21:00:09.234 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_zg_master_zone_delete ... ok 2021-03-24T21:00:09.234 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-36 2021-03-24T21:00:09.533 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-37 2021-03-24T21:00:09.590 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-38 2021-03-24T21:00:09.616 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-39 2021-03-24T21:00:11.098 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_set_bucket_website ... rgw_multi.tests.test_set_bucket_policy ... ok 2021-03-24T21:00:11.099 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-40 2021-03-24T21:00:11.125 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-41 2021-03-24T21:00:11.173 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'gtvmdf-40', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:11.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'gtvmdf-40', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:11.175 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket gtvmdf-40 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:00:11.215 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:11.215 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:11.448 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'gtvmdf-41', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:11.448 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'gtvmdf-41', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:00:11.449 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket gtvmdf-41 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:00:11.532 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:11.532 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:11.773 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:11.774 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'log', 'list', '--rgw-zone', 'test-zone1', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:11.774 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 log list --rgw-zone test-zone1 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:00:11.852 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:11.852 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:12.045 INFO:teuthology.orchestra.run.smithi052.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12:7","obj_delete_at_hint.0000000026","obj_delete_at_hint.0000000028","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4840.1-gtvmdf-35","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.3-gtvmdf-3","obj_delete_at_hint.0000000040","meta.full-sync.index.0","datalog.sync-status.shard.b3573a14-e2bf-4e98-b96a-79727832ba59.2","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","obj_delete_at_hint.0000000047","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-26:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.30:10","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17:8","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12-gtvmdf-8","obj_delete_at_hint.0000000024","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.29-gtvmdf-25","data_log.3","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.19-gtvmdf-15","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.23-gtvmdf-19","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","obj_delete_at_hint.0000000052","data.full-sync.index.b3573a14-e2bf-4e98-b96a-79727832ba59.2","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17-gtvmdf-13","2021-03-24-21-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4936.1-gtvmdf-36","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12:3","obj_delete_at_hint.0000000088","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.32-gtvmdf-28","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","meta.log.c083ad76-6371-41ed-840a-4187dd1c26c6.3","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-34:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.4:8","2021-03-24-21-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4936.6-gtvmdf-41","obj_delete_at_hint.0000000006","obj_delete_at_hint.0000000029","2021-03-24-21-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4936.5-gtvmdf-40","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12:1","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15:1","bilog.trim","obj_delete_at_hint.0000000016","2021-03-24-21-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4936.2-gtvmdf-37","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-35:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4840.1:5","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.25-gtvmdf-21","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.2-gtvmdf-32","meta.full-sync.index.1","2021-03-24-20-b3573a14-e2bf-4e98-b96a-79727832ba59.5156.1-gtvmdf-29","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.13-gtvmdf-9","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","2021-03-24-20-b3573a14-e2bf-4e98-b96a-79727832ba59.5156.2-gtvmdf-30","datalog.sync-status.shard.b3573a14-e2bf-4e98-b96a-79727832ba59.3","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12:8","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15-gtvmdf-11","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.9-gtvmdf-5","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.14-gtvmdf-10","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","datalog.sync-status.shard.b3573a14-e2bf-4e98-b96a-79727832ba59.1.retry","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15:5","datalog.sync-status.shard.b3573a14-e2bf-4e98-b96a-79727832ba59.3.retry","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-9:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.13:1","datalog.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59","mdlog.sync-status.shard.2","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.1-gtvmdf-1","data.full-sync.index.b3573a14-e2bf-4e98-b96a-79727832ba59.1","obj_delete_at_hint.0000000116","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22-gtvmdf-18","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","mdlog.sync-status.shard.1","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-33:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.3:8","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11-gtvmdf-7","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","obj_delete_at_hint.0000000066","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21:8","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.28-gtvmdf-24","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21-gtvmdf-17","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.27-gtvmdf-23","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16-gtvmdf-12","obj_delete_at_hint.0000000073","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.2-gtvmdf-2","obj_delete_at_hint.0000000038","meta.full-sync.index.3","obj_delete_at_hint.0000000118","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15:3","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.1-gtvmdf-31","obj_delete_at_hint.0000000036","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18:8","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16:5","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.24-gtvmdf-20","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.4-gtvmdf-34","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11:1","mdlog.sync-status","obj_delete_at_hint.0000000019","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11:8","obj_delete_at_hint.0000000039","obj_delete_at_hint.0000000100","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18-gtvmdf-14","2021-03-24-21-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4936.3-gtvmdf-38","obj_delete_at_hint.0000000093","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16:1","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22:8","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","obj_delete_at_hint.0000000012","datalog.sync-status.shard.b3573a14-e2bf-4e98-b96a-79727832ba59.0.retry","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.30-gtvmdf-26","2021-03-24-21-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4936.4-gtvmdf-39","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.26-gtvmdf-22","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16:3","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","datalog.sync-status.shard.b3573a14-e2bf-4e98-b96a-79727832ba59.0","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11:3","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11:7","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-10:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.14:1","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.31-gtvmdf-27","obj_delete_at_hint.0000000107","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.3-gtvmdf-33","obj_delete_at_hint.0000000046","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.4-gtvmdf-4","meta.full-sync.index.2","data_log.2","obj_delete_at_hint.0000000067","data.full-sync.index.b3573a14-e2bf-4e98-b96a-79727832ba59.0","obj_delete_at_hint.0000000086","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-16:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.20:8","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.20-gtvmdf-16","obj_delete_at_hint.0000000050","data.full-sync.index.b3573a14-e2bf-4e98-b96a-79727832ba59.3","obj_delete_at_hint.0000000102","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17:5","obj_delete_at_hint.0000000126","datalog.sync-status.shard.b3573a14-e2bf-4e98-b96a-79727832ba59.1","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-25:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.29:10","obj_delete_at_hint.0000000042","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.10-gtvmdf-6","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","datalog.sync-status.shard.b3573a14-e2bf-4e98-b96a-79727832ba59.2.retry","bucket.sync-status.b3573a14-e2bf-4e98-b96a-79727832ba59:gtvmdf-15:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.19:8"] 2021-03-24T21:00:12.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:12.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone2', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:12.072 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone2 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:00:12.108 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:12.108 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:12.314 INFO:teuthology.orchestra.run.smithi063.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-16:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.20:8","obj_delete_at_hint.0000000026","obj_delete_at_hint.0000000028","datalog.sync-status.shard.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.1.retry","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4840.1-gtvmdf-35","datalog.sync-status.shard.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.1","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.3-gtvmdf-3","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22:8","obj_delete_at_hint.0000000047","datalog.sync-status.shard.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.0","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12-gtvmdf-8","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-9:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.13:1","obj_delete_at_hint.0000000024","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-15:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.19:8","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.29-gtvmdf-25","data_log.3","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12:7","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.19-gtvmdf-15","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","data.full-sync.index.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.1","obj_delete_at_hint.0000000052","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17-gtvmdf-13","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11:1","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11:3","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-33:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.3:8","obj_delete_at_hint.0000000088","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15:3","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.32-gtvmdf-28","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","meta.log.c083ad76-6371-41ed-840a-4187dd1c26c6.3","2021-03-24-21-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4936.6-gtvmdf-41","obj_delete_at_hint.0000000006","data.full-sync.index.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.2","obj_delete_at_hint.0000000029","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bilog.trim","obj_delete_at_hint.0000000016","2021-03-24-21-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4936.2-gtvmdf-37","data.full-sync.index.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.3","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-10:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.14:1","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17:5","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16:3","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","data_log.1","datalog.sync-status.shard.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.0.retry","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.2-gtvmdf-32","meta.full-sync.index.1","2021-03-24-20-b3573a14-e2bf-4e98-b96a-79727832ba59.5156.1-gtvmdf-29","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.13-gtvmdf-9","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-34:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.4:8","2021-03-24-20-b3573a14-e2bf-4e98-b96a-79727832ba59.5156.2-gtvmdf-30","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15-gtvmdf-11","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","data.full-sync.index.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.0","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.9-gtvmdf-5","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.14-gtvmdf-10","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","mdlog.sync-status.shard.2","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.1-gtvmdf-1","obj_delete_at_hint.0000000116","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22-gtvmdf-18","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-26:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.30:10","mdlog.sync-status.shard.1","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11-gtvmdf-7","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","obj_delete_at_hint.0000000066","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.28-gtvmdf-24","datalog.sync-status.shard.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.3","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21-gtvmdf-17","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16-gtvmdf-12","obj_delete_at_hint.0000000073","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.2-gtvmdf-2","obj_delete_at_hint.0000000038","meta.full-sync.index.3","obj_delete_at_hint.0000000118","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.1-gtvmdf-31","obj_delete_at_hint.0000000036","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.24-gtvmdf-20","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.4-gtvmdf-34","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","obj_delete_at_hint.0000000039","obj_delete_at_hint.0000000100","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18-gtvmdf-14","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11:8","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16:5","obj_delete_at_hint.0000000093","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11:7","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","datalog.sync-status.shard.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.2.retry","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","data_log.0","obj_delete_at_hint.0000000012","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12:1","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.30-gtvmdf-26","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-25:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.29:10","2021-03-24-21-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4936.4-gtvmdf-39","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.26-gtvmdf-22","datalog.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18:8","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21:8","datalog.sync-status.shard.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.3.retry","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.31-gtvmdf-27","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12:3","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12:8","obj_delete_at_hint.0000000107","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.3-gtvmdf-33","obj_delete_at_hint.0000000046","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17:8","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.4-gtvmdf-4","meta.full-sync.index.2","data_log.2","obj_delete_at_hint.0000000067","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16:1","obj_delete_at_hint.0000000086","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.20-gtvmdf-16","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","datalog.sync-status.shard.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.2","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-35:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4840.1:5","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15:5","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","2021-03-24-20-662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.10-gtvmdf-6","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15:1"] 2021-03-24T21:00:12.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:12.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'log', 'list', '--rgw-zone', 'test-zone3', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:12.341 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 log list --rgw-zone test-zone3 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:00:12.419 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:12.420 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:12.615 INFO:teuthology.orchestra.run.smithi063.stdout:["obj_delete_at_hint.0000000078","meta.history","obj_delete_at_hint.0000000070","obj_delete_at_hint.0000000104","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-16:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.20:8","obj_delete_at_hint.0000000026","obj_delete_at_hint.0000000028","datalog.sync-status.shard.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.1.retry","datalog.sync-status.shard.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.1","obj_delete_at_hint.0000000040","meta.full-sync.index.0","obj_delete_at_hint.0000000015","obj_delete_at_hint.0000000069","obj_delete_at_hint.0000000095","obj_delete_at_hint.0000000003","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-18:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.22:8","obj_delete_at_hint.0000000047","datalog.sync-status.shard.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.0","obj_delete_at_hint.0000000035","obj_delete_at_hint.0000000037","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-9:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.13:1","obj_delete_at_hint.0000000024","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-15:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.19:8","obj_delete_at_hint.0000000000","obj_delete_at_hint.0000000031","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12:7","obj_delete_at_hint.0000000076","obj_delete_at_hint.0000000113","obj_delete_at_hint.0000000005","obj_delete_at_hint.0000000011","obj_delete_at_hint.0000000058","data.full-sync.index.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.1","obj_delete_at_hint.0000000052","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11:1","meta.log.ded1abf8-1b45-4199-a116-504fb1354378.0","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11:3","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-33:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.3:8","obj_delete_at_hint.0000000088","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15:3","obj_delete_at_hint.0000000080","obj_delete_at_hint.0000000090","obj_delete_at_hint.0000000110","obj_delete_at_hint.0000000096","obj_delete_at_hint.0000000087","obj_delete_at_hint.0000000008","meta.log.c083ad76-6371-41ed-840a-4187dd1c26c6.3","obj_delete_at_hint.0000000006","data.full-sync.index.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.2","obj_delete_at_hint.0000000029","meta.log.ded1abf8-1b45-4199-a116-504fb1354378.3","meta.log.ded1abf8-1b45-4199-a116-504fb1354378.2","obj_delete_at_hint.0000000089","obj_delete_at_hint.0000000022","bilog.trim","obj_delete_at_hint.0000000016","data.full-sync.index.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.3","obj_delete_at_hint.0000000041","obj_delete_at_hint.0000000018","obj_delete_at_hint.0000000092","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-10:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.14:1","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17:5","obj_delete_at_hint.0000000014","obj_delete_at_hint.0000000112","obj_delete_at_hint.0000000007","obj_delete_at_hint.0000000021","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16:3","obj_delete_at_hint.0000000064","obj_delete_at_hint.0000000071","obj_delete_at_hint.0000000074","obj_delete_at_hint.0000000081","obj_delete_at_hint.0000000009","obj_delete_at_hint.0000000121","datalog.sync-status.shard.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.0.retry","obj_delete_at_hint.0000000125","obj_delete_at_hint.0000000082","meta.full-sync.index.1","obj_delete_at_hint.0000000105","obj_delete_at_hint.0000000059","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-34:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4758.4:8","obj_delete_at_hint.0000000077","obj_delete_at_hint.0000000032","data.full-sync.index.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.0","obj_delete_at_hint.0000000053","obj_delete_at_hint.0000000091","obj_delete_at_hint.0000000065","obj_delete_at_hint.0000000083","obj_delete_at_hint.0000000010","mdlog.sync-status.shard.2","obj_delete_at_hint.0000000045","obj_delete_at_hint.0000000002","obj_delete_at_hint.0000000116","obj_delete_at_hint.0000000034","obj_delete_at_hint.0000000101","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-26:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.30:10","mdlog.sync-status.shard.1","obj_delete_at_hint.0000000079","obj_delete_at_hint.0000000049","obj_delete_at_hint.0000000117","obj_delete_at_hint.0000000044","obj_delete_at_hint.0000000066","datalog.sync-status.shard.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.3","obj_delete_at_hint.0000000068","obj_delete_at_hint.0000000085","obj_delete_at_hint.0000000073","obj_delete_at_hint.0000000038","meta.full-sync.index.3","obj_delete_at_hint.0000000118","obj_delete_at_hint.0000000036","obj_delete_at_hint.0000000103","mdlog.sync-status.shard.3","obj_delete_at_hint.0000000119","obj_delete_at_hint.0000000098","mdlog.sync-status.shard.0","obj_delete_at_hint.0000000027","mdlog.sync-status","obj_delete_at_hint.0000000019","obj_delete_at_hint.0000000039","obj_delete_at_hint.0000000100","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11:8","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16:5","obj_delete_at_hint.0000000093","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-7:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.11:7","obj_delete_at_hint.0000000004","obj_delete_at_hint.0000000063","obj_delete_at_hint.0000000122","obj_delete_at_hint.0000000057","datalog.sync-status.shard.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.2.retry","obj_delete_at_hint.0000000054","obj_delete_at_hint.0000000114","meta.log.9c8fef3d-af5e-4308-a22f-c09517bb5569.3","obj_delete_at_hint.0000000012","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12:1","obj_delete_at_hint.0000000084","obj_delete_at_hint.0000000043","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-25:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.29:10","obj_delete_at_hint.0000000111","obj_delete_at_hint.0000000048","obj_delete_at_hint.0000000020","obj_delete_at_hint.0000000099","obj_delete_at_hint.0000000056","datalog.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df","obj_delete_at_hint.0000000072","obj_delete_at_hint.0000000062","obj_delete_at_hint.0000000115","obj_delete_at_hint.0000000033","obj_delete_at_hint.0000000025","obj_delete_at_hint.0000000123","obj_delete_at_hint.0000000108","obj_delete_at_hint.0000000094","obj_delete_at_hint.0000000060","obj_delete_at_hint.0000000109","obj_delete_at_hint.0000000013","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-14:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.18:8","obj_delete_at_hint.0000000051","obj_delete_at_hint.0000000106","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-17:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.21:8","datalog.sync-status.shard.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.3.retry","obj_delete_at_hint.0000000017","obj_delete_at_hint.0000000023","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12:3","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-8:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.12:8","obj_delete_at_hint.0000000107","obj_delete_at_hint.0000000046","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-13:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.17:8","meta.full-sync.index.2","obj_delete_at_hint.0000000067","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-12:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.16:1","obj_delete_at_hint.0000000086","obj_delete_at_hint.0000000050","obj_delete_at_hint.0000000102","datalog.sync-status.shard.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.2","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-35:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4840.1:5","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15:5","obj_delete_at_hint.0000000124","obj_delete_at_hint.0000000001","obj_delete_at_hint.0000000126","obj_delete_at_hint.0000000055","obj_delete_at_hint.0000000061","obj_delete_at_hint.0000000097","obj_delete_at_hint.0000000042","obj_delete_at_hint.0000000075","obj_delete_at_hint.0000000120","obj_delete_at_hint.0000000030","bucket.sync-status.662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-11:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4277.15:1"] 2021-03-24T21:00:12.683 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:12.684 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:12.685 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:00:12.724 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:12.725 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:12.915 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616619611.745016_145.3","last_update":"2021-03-24T21:00:11.745016Z"},{"marker":"1_1616619611.745016_356.2","last_update":"2021-03-24T21:00:11.745016Z"},{"marker":"1_1616619611.745016_186.3","last_update":"2021-03-24T21:00:11.745016Z"},{"marker":"1_1616619611.745016_238.3","last_update":"2021-03-24T21:00:11.745016Z"}] 2021-03-24T21:00:12.916 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-24T21:00:12.916 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:12.917 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:12.917 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:00:12.961 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:12.961 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:13.163 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:00:13.164 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:00:18.166 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:18.167 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:18.168 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:00:18.212 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:18.213 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:18.417 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:00:18.418 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:00:23.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:23.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:23.423 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:00:23.466 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:23.466 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:23.673 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:00:23.674 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:00:28.676 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:28.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:28.677 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:00:28.722 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:28.723 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:28.936 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:00:28.936 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:00:33.938 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:33.939 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:33.940 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:00:33.982 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:33.982 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:34.185 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:00:34.186 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:00:39.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:39.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:39.190 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:00:39.235 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:39.235 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:39.450 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:00:39.451 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:00:44.453 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:44.454 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:44.454 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:00:44.499 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:44.499 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:44.719 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:00:44.719 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:00:49.721 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:49.722 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:49.722 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:00:49.769 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:49.770 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:49.997 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:00:49.998 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:00:54.999 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:55.000 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:00:55.001 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:00:55.047 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:00:55.047 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:00:55.244 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:00:55.245 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:01:00.247 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:00.247 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:00.248 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:01:00.290 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:01:00.291 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:01:00.499 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:01:00.500 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:01:05.501 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:05.502 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:05.503 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:01:05.547 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:01:05.548 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:01:05.753 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:01:05.754 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:01:10.755 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:10.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:10.756 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:01:10.800 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:01:10.800 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:01:11.010 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:01:11.011 WARNING:rgw_multi.tests:data of zone test-zone2 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:01:16.012 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:16.012 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:16.013 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:01:16.056 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:01:16.057 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:01:16.261 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619611.745016_145.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:00:11.745016Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619611.745016_356.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:00:11.745016Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619611.745016_186.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:00:11.745016Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619611.745016_238.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:00:11.745016Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:01:16.262 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-24T21:01:16.262 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:16.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:16.263 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:01:16.304 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:01:16.305 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:01:16.496 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616619611.745016_145.3","last_update":"2021-03-24T21:00:11.745016Z"},{"marker":"1_1616619611.745016_356.2","last_update":"2021-03-24T21:00:11.745016Z"},{"marker":"1_1616619611.745016_186.3","last_update":"2021-03-24T21:00:11.745016Z"},{"marker":"1_1616619611.745016_238.3","last_update":"2021-03-24T21:00:11.745016Z"}] 2021-03-24T21:01:16.496 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-24T21:01:16.497 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:16.497 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:16.498 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:01:16.541 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:01:16.541 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:01:16.797 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14224098923836610154},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:01:16.798 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:01:21.800 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:21.801 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:21.802 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:01:21.847 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:01:21.847 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:01:22.107 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14224098923836610154},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:01:22.108 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:01:27.109 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:27.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:27.110 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:01:27.152 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:01:27.153 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:01:27.413 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14224098923836610154},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:01:27.413 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:01:32.415 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:32.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:32.416 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:01:32.459 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:01:32.459 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:01:32.720 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14224098923836610154},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:01:32.721 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:01:37.722 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:37.723 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:37.723 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:01:37.766 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:01:37.767 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:01:38.025 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14224098923836610154},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:01:38.026 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:01:43.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:43.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:43.029 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:01:43.074 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:01:43.075 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:01:43.327 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14224098923836610154},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:01:43.327 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:01:48.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:48.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:48.331 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:01:48.373 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:01:48.374 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:01:48.633 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14224098923836610154},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:01:48.634 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:01:53.635 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:53.636 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:53.637 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:01:53.682 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:01:53.683 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:01:53.945 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14224098923836610154},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:01:53.946 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:01:58.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:58.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:01:58.948 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:01:58.992 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:01:58.992 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:01:59.251 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14224098923836610154},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:01:59.251 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:02:04.252 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:04.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:04.253 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:04.297 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:04.298 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:04.551 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14224098923836610154},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:02:04.551 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:02:09.553 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:09.554 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:09.556 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:09.598 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:09.599 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:09.854 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14224098923836610154},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619137.144173_40.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:52:17.144173Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619030.611627_106.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:50:30.611627Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619280.644430_130.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:54:40.644430Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619476.354833_179.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:57:56.354833Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:02:09.854 WARNING:rgw_multi.tests:data of zone test-zone3 behind zone test-zone1: shard=0 master=1_1616619611.745016_145.3 target=1_1616619137.144173_40.1, shard=1 master=1_1616619611.745016_356.2 target=1_1616619030.611627_106.2, shard=2 master=1_1616619611.745016_186.3 target=1_1616619280.644430_130.1, shard=3 master=1_1616619611.745016_238.3 target=1_1616619476.354833_179.1 2021-03-24T21:02:14.856 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:14.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:14.858 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:14.903 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:14.904 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:15.154 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14224098923836610154},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619611.745016_145.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:00:11.745016Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619611.745016_356.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:00:11.745016Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619611.745016_186.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:00:11.745016Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619611.745016_238.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:00:11.745016Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:02:15.155 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-24T21:02:15.155 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:15.156 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:15.157 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:15.196 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:15.196 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:15.388 INFO:teuthology.orchestra.run.smithi063.stdout:[{"marker":"1_1616619632.929557_149.3","last_update":"2021-03-24T21:00:32.929557Z"},{"marker":"1_1616619632.929557_359.2","last_update":"2021-03-24T21:00:32.929557Z"},{"marker":"1_1616619632.929557_220.3","last_update":"2021-03-24T21:00:32.929557Z"},{"marker":"1_1616619632.929557_257.3","last_update":"2021-03-24T21:00:32.929557Z"}] 2021-03-24T21:02:15.388 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-24T21:02:15.389 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:15.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:15.390 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:15.433 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:15.434 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:15.612 INFO:teuthology.orchestra.run.smithi052.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14589021674366829938},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619632.929557_149.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:00:32.929557Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619632.929557_359.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:00:32.929557Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619632.929557_220.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:00:32.929557Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619632.929557_257.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:00:32.929557Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:02:15.613 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-24T21:02:15.614 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable ... ok 2021-03-24T21:02:15.615 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-42 2021-03-24T21:02:15.644 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-43 2021-03-24T21:02:16.091 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-42', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:16.091 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-42', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:16.092 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-42 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:16.133 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:16.134 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:16.327 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000001.50.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.40.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.30.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.29.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:02:16.328 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-42 2021-03-24T21:02:16.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:16.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:16.331 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-42 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:16.372 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:16.372 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:16.537 INFO:teuthology.orchestra.run.smithi063.stderr:could not get bucket info for bucket=gtvmdf-42 2021-03-24T21:02:16.537 INFO:teuthology.orchestra.run.smithi063.stderr:ERROR: could not init bucket: (2) No such file or directory 2021-03-24T21:02:16.565 DEBUG:teuthology.orchestra.run:got remote process result: 2 2021-03-24T21:02:16.566 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:16.567 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:16.567 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-42 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:16.656 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:16.656 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:16.781 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:02:16.775+0000 7f2d36ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:02:16.781 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:02:16.775+0000 7f2d1eff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:02:16.781 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:02:16.775+0000 7f2d2bfff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:02:16.782 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:02:16.775+0000 7f2d22ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:02:16.782 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:02:16.775+0000 7f2d217fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:02:16.872 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:02:16.873 WARNING:rgw_multi.tests:bucket gtvmdf-42 zone test-zone2 behind zone test-zone1: shard=0 master=00000000001.50.5 target=, shard=2 master=00000000001.40.5 target=, shard=4 master=00000000001.30.5 target=, shard=7 master=00000000001.29.5 target= 2021-03-24T21:02:21.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:21.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:21.876 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-42 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:21.920 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:21.920 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:22.140 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.29.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:02:22.141 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-42 2021-03-24T21:02:22.141 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-42', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:22.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-42', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:22.142 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-42 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:22.185 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:22.186 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:22.377 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000001.50.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.40.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.30.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.29.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:02:22.377 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-42 2021-03-24T21:02:22.378 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:22.378 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:22.379 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-42 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:22.422 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:22.422 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:22.780 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.50.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.40.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.30.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.29.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:02:22.781 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-42 2021-03-24T21:02:22.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:22.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:22.783 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-42 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:22.863 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:22.864 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:23.068 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":"00000000001.35.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.70.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.42.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.27.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:02:23.069 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-42 2021-03-24T21:02:23.070 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-42', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:23.070 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-42', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:23.071 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-42 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:23.116 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:23.116 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:23.322 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.70.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.27.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:02:23.323 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-42 2021-03-24T21:02:23.323 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-42 zones={test-zone2, test-zone1} 2021-03-24T21:02:23.333 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T21:02:23.339 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-24T21:02:23.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-24T21:02:23.340 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-24T21:02:23.341 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-24T21:02:23.341 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T21:02:23.346 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-24T21:02:23.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-24T21:02:23.347 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-24T21:02:23.348 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-24T21:02:23.359 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:23.414 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:23.521 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:23.627 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:23.683 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:23.796 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:23.896 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:23.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:24.065 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:24.172 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:24.227 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:24.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:24.443 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-42 zones={test-zone2, test-zone1} 2021-03-24T21:02:24.444 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-43', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:24.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-43', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:24.445 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-43 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:24.483 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:24.483 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:24.663 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000001.29.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.42.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.23.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.39.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:02:24.664 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-43 2021-03-24T21:02:24.665 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-43', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:24.665 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-43', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:24.666 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-43 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:24.712 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:24.713 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:24.935 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.29.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.23.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:02:24.936 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-43 2021-03-24T21:02:24.937 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-43', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:24.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-43', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:24.938 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-43 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:24.982 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:24.983 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:25.173 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000001.29.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.42.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.23.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.39.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:02:25.174 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-43 2021-03-24T21:02:25.174 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-43', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:25.175 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-43', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:25.175 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-43 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:25.215 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:25.216 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:25.469 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.29.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.42.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.23.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.39.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:02:25.470 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-43 2021-03-24T21:02:25.470 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-43', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:25.471 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-43', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:25.471 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-43 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:25.551 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:25.552 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:25.755 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":"00000000001.37.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.61.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.23.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.42.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:02:25.755 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-43 2021-03-24T21:02:25.756 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-43', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:25.756 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-43', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:25.757 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-43 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:25.802 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:25.802 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:26.007 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.37.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.61.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.23.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.42.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:02:26.008 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-43 2021-03-24T21:02:26.009 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-43 zones={test-zone2, test-zone1} 2021-03-24T21:02:26.017 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T21:02:26.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-24T21:02:26.025 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-24T21:02:26.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-24T21:02:26.026 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-24T21:02:26.027 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T21:02:26.032 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-24T21:02:26.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-24T21:02:26.033 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-24T21:02:26.034 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-24T21:02:26.084 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:26.143 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:26.249 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:26.356 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:26.411 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:26.526 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:26.625 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:26.678 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:26.828 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:26.937 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:26.991 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:27.105 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:27.208 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-43 zones={test-zone2, test-zone1} 2021-03-24T21:02:27.209 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'gtvmdf-42', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:02:27.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'gtvmdf-42', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:02:27.210 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket gtvmdf-42 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:02:27.251 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:27.251 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:27.485 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'gtvmdf-42', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:02:27.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'gtvmdf-42', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:02:27.486 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket gtvmdf-42 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:02:27.562 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:27.562 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:27.798 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'gtvmdf-43', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:02:27.799 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'gtvmdf-43', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:02:27.800 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket gtvmdf-43 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:02:27.884 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:27.885 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:28.123 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'gtvmdf-43', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:02:28.123 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'gtvmdf-43', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:02:28.124 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket gtvmdf-43 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:02:28.208 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:28.208 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:28.843 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-42', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:28.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-42', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:28.844 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-42 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:28.886 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:28.887 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:29.083 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000006.56.5"},{"key":1,"val":"00000000004.37.1"},{"key":2,"val":"00000000006.52.5"},{"key":3,"val":"00000000004.24.1"},{"key":4,"val":"00000000006.40.5"},{"key":5,"val":"00000000004.51.1"},{"key":6,"val":"00000000004.29.1"},{"key":7,"val":"00000000006.35.5"},{"key":8,"val":"00000000004.45.1"},{"key":9,"val":"00000000004.38.1"},{"key":10,"val":"00000000004.27.1"}]} 2021-03-24T21:02:29.083 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-42 2021-03-24T21:02:29.084 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:29.084 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:29.085 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-42 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:29.130 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:29.131 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:29.338 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.30.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.29.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:02:29.339 WARNING:rgw_multi.tests:bucket gtvmdf-42 zone test-zone2 behind zone test-zone1: shard=0 master=00000000006.56.5 target=00000000001.50.5, shard=1 master=00000000004.37.1 target=, shard=2 master=00000000006.52.5 target=00000000001.40.5, shard=3 master=00000000004.24.1 target=, shard=4 master=00000000006.40.5 target=00000000001.30.5, shard=5 master=00000000004.51.1 target=, shard=6 master=00000000004.29.1 target=, shard=7 master=00000000006.35.5 target=00000000001.29.5, shard=8 master=00000000004.45.1 target=, shard=9 master=00000000004.38.1 target=, shard=10 master=00000000004.27.1 target= 2021-03-24T21:02:34.341 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:34.342 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:34.342 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-42 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:34.387 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:34.388 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:34.598 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.56.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.37.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.52.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.24.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.40.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":3},"inc_marker":{"position":"00000000006.35.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.38.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.27.1"}}}] 2021-03-24T21:02:34.599 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-42 2021-03-24T21:02:34.599 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-42', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:34.600 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-42', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:34.600 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-42 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:34.643 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:34.644 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:34.837 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000006.56.5"},{"key":1,"val":"00000000004.37.1"},{"key":2,"val":"00000000006.52.5"},{"key":3,"val":"00000000004.24.1"},{"key":4,"val":"00000000006.40.5"},{"key":5,"val":"00000000004.51.1"},{"key":6,"val":"00000000004.29.1"},{"key":7,"val":"00000000006.35.5"},{"key":8,"val":"00000000004.45.1"},{"key":9,"val":"00000000004.38.1"},{"key":10,"val":"00000000004.27.1"}]} 2021-03-24T21:02:34.837 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-42 2021-03-24T21:02:34.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:34.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:34.839 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-42 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:34.885 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:34.886 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:35.144 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.56.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.37.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.52.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.24.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.40.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.51.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.35.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.45.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.38.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.27.1"}}}] 2021-03-24T21:02:35.145 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-42 2021-03-24T21:02:35.146 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:35.147 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-42', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:35.148 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-42 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:35.223 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:35.224 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:35.410 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":"00000000002.37.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.74.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.44.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.31.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:02:35.410 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-42 2021-03-24T21:02:35.411 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-42', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:35.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-42', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:35.412 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-42 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:35.455 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:35.456 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:35.656 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.37.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.74.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.44.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.31.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:02:35.657 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-42 2021-03-24T21:02:35.657 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-42 zones={test-zone2, test-zone1} 2021-03-24T21:02:35.666 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T21:02:35.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-24T21:02:35.672 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-24T21:02:35.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-24T21:02:35.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-24T21:02:35.673 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-24T21:02:35.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-24T21:02:35.674 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-24T21:02:35.675 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-24T21:02:35.675 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T21:02:35.681 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-24T21:02:35.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-24T21:02:35.682 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-24T21:02:35.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-24T21:02:35.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-24T21:02:35.683 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-24T21:02:35.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-24T21:02:35.684 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-24T21:02:35.694 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:35.751 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:35.856 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:35.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:36.025 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:36.144 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:36.252 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:36.308 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:36.418 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:36.524 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:36.584 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:36.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:36.797 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-24T21:02:36.851 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-24T21:02:36.964 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-24T21:02:37.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-24T21:02:37.124 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-24T21:02:37.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-24T21:02:37.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-24T21:02:37.391 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-24T21:02:37.496 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-24T21:02:37.595 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-24T21:02:37.655 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-24T21:02:37.770 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-24T21:02:37.872 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-42 zones={test-zone2, test-zone1} 2021-03-24T21:02:37.873 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-43', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:37.874 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-43', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:37.874 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-43 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:37.916 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:37.917 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:38.110 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000006.35.5"},{"key":1,"val":"00000000004.24.1"},{"key":2,"val":"00000000006.54.5"},{"key":3,"val":"00000000004.31.1"},{"key":4,"val":"00000000006.33.5"},{"key":5,"val":"00000000004.42.1"},{"key":6,"val":"00000000004.32.1"},{"key":7,"val":"00000000006.49.5"},{"key":8,"val":"00000000004.62.1"},{"key":9,"val":"00000000004.21.1"},{"key":10,"val":"00000000004.47.1"}]} 2021-03-24T21:02:38.111 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-43 2021-03-24T21:02:38.111 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-43', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:38.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-43', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:38.112 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-43 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:38.156 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:38.156 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:38.358 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.35.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.24.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.54.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.31.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.33.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.32.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.49.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.21.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}}] 2021-03-24T21:02:38.359 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-43 2021-03-24T21:02:38.359 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-43', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:38.360 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-43', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:38.361 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-43 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:38.401 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:38.402 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:38.588 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000006.35.5"},{"key":1,"val":"00000000004.24.1"},{"key":2,"val":"00000000006.54.5"},{"key":3,"val":"00000000004.31.1"},{"key":4,"val":"00000000006.33.5"},{"key":5,"val":"00000000004.42.1"},{"key":6,"val":"00000000004.32.1"},{"key":7,"val":"00000000006.49.5"},{"key":8,"val":"00000000004.62.1"},{"key":9,"val":"00000000004.21.1"},{"key":10,"val":"00000000004.47.1"}]} 2021-03-24T21:02:38.589 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-43 2021-03-24T21:02:38.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-43', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:38.590 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-43', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:38.590 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-43 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:38.635 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:38.636 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:38.993 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.35.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.24.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.54.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.31.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.33.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.42.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.32.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.49.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.62.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.21.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.47.1"}}}] 2021-03-24T21:02:38.994 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-43 2021-03-24T21:02:38.994 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-43', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:38.995 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-43', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:38.995 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-43 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:39.077 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:39.077 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:39.263 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":"00000000002.40.5"},{"key":1,"val":""},{"key":2,"val":"00000000002.72.5"},{"key":3,"val":""},{"key":4,"val":"00000000002.32.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000002.46.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:02:39.264 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-43 2021-03-24T21:02:39.264 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-43', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:39.265 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-43', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:39.265 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-43 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:39.309 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:39.309 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:39.510 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.40.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.72.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.32.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000002.46.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:02:39.511 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-43 2021-03-24T21:02:39.512 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-43 zones={test-zone2, test-zone1} 2021-03-24T21:02:39.521 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T21:02:39.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-24T21:02:39.527 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-24T21:02:39.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-24T21:02:39.528 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-24T21:02:39.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-24T21:02:39.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-24T21:02:39.529 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-24T21:02:39.530 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-24T21:02:39.530 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T21:02:39.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-24T21:02:39.537 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-24T21:02:39.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-24T21:02:39.538 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-24T21:02:39.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-24T21:02:39.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-24T21:02:39.539 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-24T21:02:39.540 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-24T21:02:39.552 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:39.606 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:39.717 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:39.824 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:39.883 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:39.994 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:40.139 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:40.195 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:40.306 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:40.417 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:40.471 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:40.586 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:40.696 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-24T21:02:40.757 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-24T21:02:40.870 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-24T21:02:40.973 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-24T21:02:41.027 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-24T21:02:41.138 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-24T21:02:41.236 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-24T21:02:41.291 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-24T21:02:41.403 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-24T21:02:41.509 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-24T21:02:41.565 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-24T21:02:41.673 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-24T21:02:41.777 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-43 zones={test-zone2, test-zone1} 2021-03-24T21:02:41.777 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:41.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:41.781 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:41.819 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:41.820 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:42.012 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616619748.420555_168.3","last_update":"2021-03-24T21:02:28.420555Z"},{"marker":"1_1616619748.420555_384.2","last_update":"2021-03-24T21:02:28.420555Z"},{"marker":"1_1616619748.420555_227.3","last_update":"2021-03-24T21:02:28.420555Z"},{"marker":"1_1616619748.420555_270.3","last_update":"2021-03-24T21:02:28.420555Z"}] 2021-03-24T21:02:42.012 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-24T21:02:42.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:42.013 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:42.014 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:42.059 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:42.060 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:42.251 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619748.420555_168.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:02:28.420555Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619748.420555_384.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:02:28.420555Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619748.420555_227.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:02:28.420555Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619748.420555_270.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:02:28.420555Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:02:42.252 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-24T21:02:42.253 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:42.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:42.254 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:42.296 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:42.296 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:42.484 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616619748.420555_168.3","last_update":"2021-03-24T21:02:28.420555Z"},{"marker":"1_1616619748.420555_384.2","last_update":"2021-03-24T21:02:28.420555Z"},{"marker":"1_1616619748.420555_227.3","last_update":"2021-03-24T21:02:28.420555Z"},{"marker":"1_1616619748.420555_270.3","last_update":"2021-03-24T21:02:28.420555Z"}] 2021-03-24T21:02:42.484 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-24T21:02:42.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:42.485 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:42.486 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:42.529 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:42.530 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:42.788 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14224098923836610154},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619748.420555_168.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:02:28.420555Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619748.420555_384.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:02:28.420555Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619748.420555_227.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:02:28.420555Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619748.420555_270.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:02:28.420555Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:02:42.789 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-24T21:02:42.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:42.790 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:42.791 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:42.869 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:42.869 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:43.066 INFO:teuthology.orchestra.run.smithi063.stdout:[{"marker":"1_1616619736.968901_154.1","last_update":"2021-03-24T21:02:16.968901Z"},{"marker":"1_1616619737.983199_374.1","last_update":"2021-03-24T21:02:17.983199Z"},{"marker":"1_1616619736.975092_240.1","last_update":"2021-03-24T21:02:16.975092Z"},{"marker":"1_1616619632.929557_257.3","last_update":"2021-03-24T21:00:32.929557Z"}] 2021-03-24T21:02:43.066 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-24T21:02:43.067 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:43.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:43.068 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:43.111 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:43.111 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:43.310 INFO:teuthology.orchestra.run.smithi052.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14589021674366829938},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619736.968901_154.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:02:16.968901Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619737.983199_374.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:02:17.983199Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619736.975092_240.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:02:16.975092Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619632.929557_257.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:00:32.929557Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:02:43.311 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-24T21:02:43.312 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_enable_right_after_disable ... ok 2021-03-24T21:02:43.312 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-44 2021-03-24T21:02:43.340 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-45 2021-03-24T21:02:43.783 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:43.784 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:43.785 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:43.827 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:43.828 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:43.994 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:02:44.025 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616619763.370238_374.1","last_update":"2021-03-24T21:02:43.370238Z"}] 2021-03-24T21:02:44.025 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:02:44.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:44.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:44.027 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:44.068 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:44.069 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:44.260 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619748.411470_366.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:02:28.411470Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:02:44.261 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616619763.370238_374.1 target=1_1616619748.411470_366.1 2021-03-24T21:02:49.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:49.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:49.264 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:49.306 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:49.306 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:49.498 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619763.370238_374.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:02:43.370238Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:02:49.499 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:02:49.499 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:02:49.500 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:49.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:49.501 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:49.585 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:49.585 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:49.847 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619763.370238_374.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:02:43.370238Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:02:49.848 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:02:49.849 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:49.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:49.850 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-44 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:49.893 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:49.893 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:50.080 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000001.26.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.39.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.20.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.53.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:02:50.080 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-44 2021-03-24T21:02:50.081 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:50.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:50.082 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:50.127 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:50.127 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:50.335 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.26.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.20.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:02:50.336 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-44 2021-03-24T21:02:50.337 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:50.337 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:50.338 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-44 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:50.382 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:50.383 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:50.573 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000001.26.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.39.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.20.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.53.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:02:50.574 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-44 2021-03-24T21:02:50.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:50.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:50.575 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:50.621 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:50.622 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:50.886 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.26.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.39.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.20.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.53.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:02:50.887 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-44 2021-03-24T21:02:50.887 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:50.888 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:50.888 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:50.969 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:50.970 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:51.153 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":"00000000001.35.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.48.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.21.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.55.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:02:51.153 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-44 2021-03-24T21:02:51.154 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:51.155 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:51.155 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-44 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:51.200 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:51.200 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:51.408 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.48.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.21.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.55.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:02:51.408 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-44 2021-03-24T21:02:51.409 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-44 zones={test-zone2, test-zone1} 2021-03-24T21:02:51.418 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T21:02:51.467 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-24T21:02:51.467 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-24T21:02:51.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-24T21:02:51.468 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-24T21:02:51.469 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T21:02:51.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-24T21:02:51.475 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-24T21:02:51.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-24T21:02:51.476 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-24T21:02:51.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:51.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:51.697 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:51.801 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:51.855 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:51.962 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:52.069 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:52.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:52.229 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:52.329 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:52.383 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:52.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:52.587 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-44 zones={test-zone2, test-zone1} 2021-03-24T21:02:52.588 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-45', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:52.589 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-45', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:52.589 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-45 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:52.631 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:52.632 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:52.832 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000001.40.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.44.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.40.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.49.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:02:52.832 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-45 2021-03-24T21:02:52.833 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-45', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:52.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-45', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:52.834 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-45 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:52.879 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:52.879 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:53.099 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.44.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.40.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:02:53.100 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-45 2021-03-24T21:02:53.101 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-45', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:53.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-45', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:53.102 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-45 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:53.144 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:53.144 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:53.351 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000001.40.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.44.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.40.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.49.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:02:53.352 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-45 2021-03-24T21:02:53.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-45', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:53.353 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-45', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:53.354 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-45 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:53.398 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:53.399 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:53.656 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.40.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.44.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.40.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.49.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:02:53.657 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-45 2021-03-24T21:02:53.658 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-45', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:53.658 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-45', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:53.659 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-45 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:53.740 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:53.741 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:53.935 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":"00000000001.49.5"},{"key":1,"val":""},{"key":2,"val":"00000000001.52.5"},{"key":3,"val":""},{"key":4,"val":"00000000001.50.5"},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":"00000000001.63.5"},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:02:53.936 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-45 2021-03-24T21:02:53.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-45', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:53.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-45', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:53.937 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-45 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:53.980 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:53.981 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:54.198 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.49.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:02:54.199 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-45 2021-03-24T21:02:54.199 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-45 zones={test-zone2, test-zone1} 2021-03-24T21:02:54.208 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T21:02:54.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-24T21:02:54.259 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-24T21:02:54.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-24T21:02:54.260 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-24T21:02:54.261 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T21:02:54.267 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-24T21:02:54.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-24T21:02:54.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-24T21:02:54.268 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-24T21:02:54.320 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:54.382 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:54.488 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:02:54.596 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:54.651 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:54.761 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:02:54.860 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:54.915 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:55.030 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:02:55.136 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:55.190 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:55.297 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:02:55.400 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-45 zones={test-zone2, test-zone1} 2021-03-24T21:02:55.401 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'gtvmdf-44', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:02:55.402 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'gtvmdf-44', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:02:55.402 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket gtvmdf-44 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:02:55.447 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:55.447 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:55.675 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'disable', '--bucket', 'gtvmdf-45', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:02:55.676 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'disable', '--bucket', 'gtvmdf-45', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:02:55.676 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync disable --bucket gtvmdf-45 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:02:55.758 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:55.759 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:56.006 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:56.007 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:56.008 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:56.092 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:56.093 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:56.267 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:02:56.298 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616619775.969120_378.1","last_update":"2021-03-24T21:02:55.969120Z"}] 2021-03-24T21:02:56.299 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:02:56.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:56.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:02:56.300 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:02:56.343 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:02:56.344 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:02:56.520 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619763.370238_374.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:02:43.370238Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:02:56.521 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616619775.969120_378.1 target=1_1616619763.370238_374.1 2021-03-24T21:03:01.523 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:01.523 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:01.524 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:03:01.565 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:01.565 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:01.770 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619775.969120_378.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:02:55.969120Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:03:01.771 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:03:01.772 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:03:01.772 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:01.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:01.773 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:03:01.852 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:01.853 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:02.113 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619775.969120_378.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:02:55.969120Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:03:02.114 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:03:02.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'gtvmdf-44', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:03:02.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'gtvmdf-44', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:03:02.508 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket gtvmdf-44 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:03:02.551 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:02.551 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:02.795 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'enable', '--bucket', 'gtvmdf-45', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:03:02.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'enable', '--bucket', 'gtvmdf-45', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:03:02.796 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync enable --bucket gtvmdf-45 --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:03:02.881 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:02.882 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:03.131 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:03.132 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:03.132 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-44 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:03:03.215 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:03.215 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:03.386 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000006.32.1"},{"key":1,"val":"00000000004.56.1"},{"key":2,"val":"00000000006.48.1"},{"key":3,"val":"00000000004.35.1"},{"key":4,"val":"00000000006.28.1"},{"key":5,"val":"00000000004.49.1"},{"key":6,"val":"00000000004.55.1"},{"key":7,"val":"00000000006.61.1"},{"key":8,"val":"00000000004.43.1"},{"key":9,"val":"00000000004.29.1"},{"key":10,"val":"00000000004.35.1"}]} 2021-03-24T21:03:03.386 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-44 2021-03-24T21:03:03.387 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:03.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:03.388 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:03:03.427 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:03.427 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:03.596 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.32.1"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.48.1"}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.28.1"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:03:03.596 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.56.1 target=, shard=3 master=00000000004.35.1 target=, shard=5 master=00000000004.49.1 target=, shard=7 master=00000000006.61.1 target=, shard=9 master=00000000004.29.1 target= 2021-03-24T21:03:08.598 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:08.599 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:08.600 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:03:08.646 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:08.646 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:08.872 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.32.1"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.48.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.28.1"}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.61.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:03:08.873 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone2 behind zone test-zone1: shard=1 master=00000000004.56.1 target=, shard=5 master=00000000004.49.1 target=, shard=9 master=00000000004.29.1 target= 2021-03-24T21:03:13.874 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:13.875 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:13.876 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:03:13.922 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:13.923 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:14.144 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.32.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.48.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.28.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000006.61.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:03:14.145 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-44 2021-03-24T21:03:14.145 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:14.146 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:14.147 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-44 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:03:14.188 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:14.189 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:14.394 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000006.32.1"},{"key":1,"val":"00000000004.56.1"},{"key":2,"val":"00000000006.48.1"},{"key":3,"val":"00000000004.35.1"},{"key":4,"val":"00000000006.28.1"},{"key":5,"val":"00000000004.49.1"},{"key":6,"val":"00000000004.55.1"},{"key":7,"val":"00000000006.61.1"},{"key":8,"val":"00000000004.43.1"},{"key":9,"val":"00000000004.29.1"},{"key":10,"val":"00000000004.35.1"}]} 2021-03-24T21:03:14.394 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-44 2021-03-24T21:03:14.395 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:14.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:14.396 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:03:14.440 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:14.440 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:14.701 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:03:14.702 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:03:19.704 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:19.705 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:19.706 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:03:19.751 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:19.752 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:20.011 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:03:20.012 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:03:25.013 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:25.014 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:25.015 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:03:25.059 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:25.059 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:25.426 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:03:25.426 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:03:30.428 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:30.429 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:30.430 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:03:30.475 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:30.475 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:30.735 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:03:30.736 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:03:35.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:35.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:35.739 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:03:35.786 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:35.786 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:36.046 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:03:36.047 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:03:41.048 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:41.049 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:41.049 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:03:41.096 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:41.097 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:41.353 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:03:41.353 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:03:46.354 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:46.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:46.356 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:03:46.400 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:46.400 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:46.660 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:03:46.661 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:03:51.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:51.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:51.664 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:03:51.708 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:51.709 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:51.970 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:03:51.971 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:03:56.972 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:56.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:03:56.973 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:03:57.019 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:03:57.019 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:03:57.278 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:03:57.279 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:04:02.280 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:02.281 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:02.281 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:04:02.331 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:04:02.331 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:04:02.592 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:04:02.643 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:04:07.645 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:07.646 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:07.646 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:04:07.692 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:04:07.692 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:04:07.951 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:04:07.952 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:04:12.953 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:12.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:12.955 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:04:13.001 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:04:13.002 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:04:13.256 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:04:13.257 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:04:18.259 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:18.260 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:18.261 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:04:18.306 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:04:18.306 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:04:18.560 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:04:18.561 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:04:23.562 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:23.563 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:23.564 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:04:23.609 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:04:23.609 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:04:23.870 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:04:23.871 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:04:28.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:28.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:28.873 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:04:28.917 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:04:28.918 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:04:29.176 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:04:29.177 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:04:34.178 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:34.179 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:34.180 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:04:34.227 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:04:34.227 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:04:34.482 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:04:34.483 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:04:39.485 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:39.486 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:39.486 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:04:39.533 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:04:39.533 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:04:39.789 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:04:39.790 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:04:44.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:44.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:44.793 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:04:44.838 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:04:44.838 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:04:45.098 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:04:45.098 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:04:50.100 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:50.101 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:50.102 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:04:50.147 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:04:50.147 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:04:50.410 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:04:50.410 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:04:55.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:55.412 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:04:55.413 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:04:55.459 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:04:55.459 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:04:55.718 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:04:55.718 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:05:00.720 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:00.720 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:00.721 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:00.767 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:00.767 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:01.026 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:05:01.027 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:05:06.028 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:06.029 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:06.031 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:06.076 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:06.077 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:06.333 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:05:06.334 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:05:11.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:11.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:11.337 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:11.382 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:11.383 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:11.641 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:05:11.642 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:05:16.644 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:16.645 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:16.645 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:16.691 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:16.692 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:16.950 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:05:16.950 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:05:21.951 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:21.952 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:21.953 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:21.996 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:21.997 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:22.259 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:05:22.260 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:05:27.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:27.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:27.263 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:27.308 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:27.308 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:27.561 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.27.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.21.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"stopped","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:05:27.562 WARNING:rgw_multi.tests:bucket gtvmdf-44 zone test-zone3 behind zone test-zone1: shard=0 master=00000000006.32.1 target=00000000002.27.1, shard=2 master=00000000006.48.1 target=00000000002.40.1, shard=4 master=00000000006.28.1 target=00000000002.21.1, shard=7 master=00000000006.61.1 target=00000000002.54.1 2021-03-24T21:05:32.563 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:32.564 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:32.565 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:32.613 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:32.613 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:32.876 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.32.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.56.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.48.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.28.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.49.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.55.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.61.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.43.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.29.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.35.1"}}}] 2021-03-24T21:05:32.877 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-44 2021-03-24T21:05:32.878 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:32.878 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-44', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:32.879 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-44 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:32.920 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:32.921 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:33.134 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":"00000000004.40.1"},{"key":1,"val":"00000000002.52.1"},{"key":2,"val":"00000000004.55.1"},{"key":3,"val":"00000000002.48.1"},{"key":4,"val":"00000000004.29.1"},{"key":5,"val":"00000000002.56.1"},{"key":6,"val":"00000000002.73.1"},{"key":7,"val":"00000000004.63.1"},{"key":8,"val":"00000000002.45.1"},{"key":9,"val":"00000000002.25.1"},{"key":10,"val":"00000000002.36.1"}]} 2021-03-24T21:05:33.135 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-44 2021-03-24T21:05:33.136 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:33.136 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-44', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:33.137 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-44 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:33.182 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:33.183 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:33.402 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.40.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.52.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.55.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.48.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.29.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.56.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.73.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.63.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.45.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.25.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.36.1"}}}] 2021-03-24T21:05:33.402 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-44 2021-03-24T21:05:33.403 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-44 zones={test-zone2, test-zone1} 2021-03-24T21:05:33.416 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T21:05:33.422 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-24T21:05:33.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-24T21:05:33.423 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-24T21:05:33.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-24T21:05:33.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-24T21:05:33.424 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-24T21:05:33.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-24T21:05:33.425 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-24T21:05:33.426 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T21:05:33.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-24T21:05:33.433 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-24T21:05:33.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-24T21:05:33.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-24T21:05:33.434 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-24T21:05:33.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-24T21:05:33.435 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-24T21:05:33.436 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-24T21:05:33.446 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:05:33.499 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:05:33.612 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:05:33.720 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:05:33.775 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:05:33.886 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:05:33.988 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:05:34.043 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:05:34.158 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:05:34.264 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:05:34.318 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:05:34.426 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:05:34.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-24T21:05:34.583 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-24T21:05:34.691 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-24T21:05:34.888 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-24T21:05:34.947 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-24T21:05:35.060 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-24T21:05:35.168 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-24T21:05:35.223 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-24T21:05:35.333 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-24T21:05:35.440 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-24T21:05:35.495 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-24T21:05:35.600 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-24T21:05:35.708 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-44 zones={test-zone2, test-zone1} 2021-03-24T21:05:35.709 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-45', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:35.710 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-45', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:35.710 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-45 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:35.751 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:35.751 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:35.952 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000006.46.5"},{"key":1,"val":"00000000004.63.1"},{"key":2,"val":"00000000006.54.5"},{"key":3,"val":"00000000004.30.1"},{"key":4,"val":"00000000006.46.5"},{"key":5,"val":"00000000004.34.1"},{"key":6,"val":"00000000004.105.1"},{"key":7,"val":"00000000006.59.5"},{"key":8,"val":"00000000004.52.1"},{"key":9,"val":"00000000004.40.1"},{"key":10,"val":"00000000004.37.1"}]} 2021-03-24T21:05:35.953 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-45 2021-03-24T21:05:35.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-45', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:35.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-45', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:35.955 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-45 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:35.999 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:36.000 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:36.227 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj3","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.46.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.63.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj2","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.54.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj1","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.46.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.34.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.105.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj4","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000006.59.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.40.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.37.1"}}}] 2021-03-24T21:05:36.228 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-45 2021-03-24T21:05:36.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-45', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:36.229 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-45', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:36.230 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-45 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:36.272 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:36.273 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:36.474 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000006.46.5"},{"key":1,"val":"00000000004.63.1"},{"key":2,"val":"00000000006.54.5"},{"key":3,"val":"00000000004.30.1"},{"key":4,"val":"00000000006.46.5"},{"key":5,"val":"00000000004.34.1"},{"key":6,"val":"00000000004.105.1"},{"key":7,"val":"00000000006.59.5"},{"key":8,"val":"00000000004.52.1"},{"key":9,"val":"00000000004.40.1"},{"key":10,"val":"00000000004.37.1"}]} 2021-03-24T21:05:36.475 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-45 2021-03-24T21:05:36.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-45', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:36.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-45', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:36.477 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-45 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:36.520 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:36.520 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:36.777 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.63.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.54.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.30.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.46.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.34.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.105.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.59.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.52.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.40.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.37.1"}}}] 2021-03-24T21:05:36.778 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-45 2021-03-24T21:05:36.778 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-45', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:36.779 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-45', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:36.780 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-45 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:36.862 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:36.862 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:37.073 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":"00000000004.55.1"},{"key":1,"val":"00000000002.62.1"},{"key":2,"val":"00000000004.61.1"},{"key":3,"val":"00000000002.28.1"},{"key":4,"val":"00000000004.56.1"},{"key":5,"val":"00000000002.40.1"},{"key":6,"val":"00000000002.112.1"},{"key":7,"val":"00000000004.72.1"},{"key":8,"val":"00000000002.62.1"},{"key":9,"val":"00000000002.35.1"},{"key":10,"val":"00000000002.34.1"}]} 2021-03-24T21:05:37.073 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-45 2021-03-24T21:05:37.074 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-45', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:37.074 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-45', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:37.075 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-45 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:37.121 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:37.122 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:37.341 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj7","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.55.1"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.62.1"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj6","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.61.1"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.28.1"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj5","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.56.1"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.40.1"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.112.1"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"obj8","instance":"","ns":""},"count":2},"inc_marker":{"position":"00000000004.72.1"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.62.1"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.35.1"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.34.1"}}}] 2021-03-24T21:05:37.342 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-45 2021-03-24T21:05:37.343 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-45 zones={test-zone2, test-zone1} 2021-03-24T21:05:37.352 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T21:05:37.357 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-24T21:05:37.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-24T21:05:37.358 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-24T21:05:37.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-24T21:05:37.359 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-24T21:05:37.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-24T21:05:37.360 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-24T21:05:37.361 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-24T21:05:37.361 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T21:05:37.366 DEBUG:tasks.rgw_multi.zone_rados:o=obj1 2021-03-24T21:05:37.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj2 2021-03-24T21:05:37.367 DEBUG:tasks.rgw_multi.zone_rados:o=obj3 2021-03-24T21:05:37.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj4 2021-03-24T21:05:37.368 DEBUG:tasks.rgw_multi.zone_rados:o=obj5 2021-03-24T21:05:37.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj6 2021-03-24T21:05:37.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj7 2021-03-24T21:05:37.369 DEBUG:tasks.rgw_multi.zone_rados:o=obj8 2021-03-24T21:05:37.379 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:05:37.431 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:05:37.533 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj1 2021-03-24T21:05:37.640 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:05:37.695 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:05:37.802 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj2 2021-03-24T21:05:37.900 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:05:37.955 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:05:38.100 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj3 2021-03-24T21:05:38.204 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:05:38.263 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:05:38.369 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj4 2021-03-24T21:05:38.472 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-24T21:05:38.528 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-24T21:05:38.642 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj5 2021-03-24T21:05:38.745 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-24T21:05:38.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-24T21:05:38.905 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj6 2021-03-24T21:05:39.012 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-24T21:05:39.067 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-24T21:05:39.185 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj7 2021-03-24T21:05:39.288 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-24T21:05:39.343 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-24T21:05:39.453 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=obj8 2021-03-24T21:05:39.556 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-45 zones={test-zone2, test-zone1} 2021-03-24T21:05:39.557 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:39.558 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:39.558 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:39.597 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:39.598 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:39.789 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616619807.442472_213.1","last_update":"2021-03-24T21:03:27.442472Z"},{"marker":"1_1616619807.442472_432.1","last_update":"2021-03-24T21:03:27.442472Z"},{"marker":"1_1616619807.442472_277.2","last_update":"2021-03-24T21:03:27.442472Z"},{"marker":"1_1616619783.103201_307.3","last_update":"2021-03-24T21:03:03.103201Z"}] 2021-03-24T21:05:39.790 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-24T21:05:39.790 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:39.791 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:39.791 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:39.834 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:39.835 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:40.054 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":111494104620731188},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619807.442472_213.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:03:27.442472Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619807.442472_432.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:03:27.442472Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619807.442472_277.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:03:27.442472Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619783.103201_307.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:03:03.103201Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:05:40.054 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone2 source_zone=test-zone1 2021-03-24T21:05:40.055 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:40.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'datalog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:40.056 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 datalog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:40.095 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:40.095 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:40.295 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"1_1616619807.442472_213.1","last_update":"2021-03-24T21:03:27.442472Z"},{"marker":"1_1616619807.442472_432.1","last_update":"2021-03-24T21:03:27.442472Z"},{"marker":"1_1616619807.442472_277.2","last_update":"2021-03-24T21:03:27.442472Z"},{"marker":"1_1616619783.103201_307.3","last_update":"2021-03-24T21:03:03.103201Z"}] 2021-03-24T21:05:40.296 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-24T21:05:40.296 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:40.297 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'data', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:40.298 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 data sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:40.342 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:40.342 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:40.602 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14224098923836610154},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619807.442472_213.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:03:27.442472Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619807.442472_432.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:03:27.442472Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619807.442472_277.2","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:03:27.442472Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619783.103201_307.3","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:03:03.103201Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:05:40.603 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone3 source_zone=test-zone1 2021-03-24T21:05:40.603 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:40.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'datalog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:40.604 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 datalog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:40.684 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:40.684 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:40.901 INFO:teuthology.orchestra.run.smithi063.stdout:[{"marker":"1_1616619786.955177_177.6","last_update":"2021-03-24T21:03:06.955177Z"},{"marker":"1_1616619786.955177_404.4","last_update":"2021-03-24T21:03:06.955177Z"},{"marker":"1_1616619786.955177_295.6","last_update":"2021-03-24T21:03:06.955177Z"},{"marker":"1_1616619786.955177_325.6","last_update":"2021-03-24T21:03:06.955177Z"}] 2021-03-24T21:05:40.901 INFO:rgw_multi.tests:starting data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-24T21:05:40.902 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:40.902 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'data', 'sync', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:40.903 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 data sync status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:40.947 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:40.947 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:41.151 INFO:teuthology.orchestra.run.smithi052.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"instance_id":14589021674366829938},"markers":[{"key":0,"val":{"status":"incremental-sync","marker":"1_1616619786.955177_177.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:03:06.955177Z"}},{"key":1,"val":{"status":"incremental-sync","marker":"1_1616619786.955177_404.4","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:03:06.955177Z"}},{"key":2,"val":{"status":"incremental-sync","marker":"1_1616619786.955177_295.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:03:06.955177Z"}},{"key":3,"val":{"status":"incremental-sync","marker":"1_1616619786.955177_325.6","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:03:06.955177Z"}}]},"full_sync":{"total":0,"complete":0}} 2021-03-24T21:05:41.152 INFO:rgw_multi.tests:finished data checkpoint for target_zone=test-zone1 source_zone=test-zone2 2021-03-24T21:05:41.153 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_sync_disable_enable ... ok 2021-03-24T21:05:41.153 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-46 2021-03-24T21:05:41.181 INFO:rgw_multi.tests:create bucket zone=test-zone2 name=gtvmdf-47 2021-03-24T21:05:43.171 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:43.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:43.172 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-46 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:43.214 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:43.214 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:43.414 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.49.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:05:43.415 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-46 2021-03-24T21:05:43.416 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-46', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:43.416 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-46', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:43.417 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-46 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:43.460 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:43.460 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:43.684 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:05:43.685 WARNING:rgw_multi.tests:bucket gtvmdf-46 zone test-zone2 behind zone test-zone1: shard=5 master=00000000011.49.5 target= 2021-03-24T21:05:48.686 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-46', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:48.687 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-46', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:48.688 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-46 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:48.732 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:48.732 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:48.971 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000011.49.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:05:48.972 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-46 2021-03-24T21:05:48.972 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:48.973 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:48.974 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-46 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:49.017 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:49.018 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:49.220 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000011.49.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:05:49.220 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-46 2021-03-24T21:05:49.221 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-46', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:49.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-46', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:49.222 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-46 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:49.267 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:49.268 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:49.528 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000011.49.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:05:49.528 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-46 2021-03-24T21:05:49.529 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-46', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:49.530 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-46', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:49.530 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-46 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:49.572 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:49.573 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:49.791 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.45.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:05:49.791 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-46 2021-03-24T21:05:49.792 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:49.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-46', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:49.793 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-46 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:49.838 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:49.838 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:50.048 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"MULTIPART","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:05:50.049 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-46 2021-03-24T21:05:50.049 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-46 zones={test-zone2, test-zone1} 2021-03-24T21:05:50.058 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T21:05:50.107 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-03-24T21:05:50.107 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T21:05:50.113 DEBUG:tasks.rgw_multi.zone_rados:o=MULTIPART 2021-03-24T21:05:50.164 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-24T21:05:50.366 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-24T21:05:50.598 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=MULTIPART 2021-03-24T21:05:50.835 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-46 zones={test-zone2, test-zone1} 2021-03-24T21:05:50.835 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_multipart_object_sync ... ok 2021-03-24T21:05:50.836 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-48 2021-03-24T21:05:50.959 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:50.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:50.960 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:51.000 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:51.002 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:51.169 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:05:51.201 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616619950.865493_394.1","last_update":"2021-03-24T21:05:50.865493Z"}] 2021-03-24T21:05:51.201 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:05:51.202 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:51.203 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:51.203 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:51.244 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:51.245 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:51.447 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619950.865493_394.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:05:50.865493Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:05:51.448 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:05:51.448 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:05:51.449 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:51.449 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:51.450 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:51.532 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:51.533 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:51.790 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619941.211348_390.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:05:41.211348Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:05:51.790 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1616619950.865493_394.1 target=1_1616619941.211348_390.1 2021-03-24T21:05:51.933 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:05:51.929+0000 7f442a7ec700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:51.933 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:05:51.929+0000 7f442c7f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:51.934 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:05:51.929+0000 7f44467fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:51.934 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:05:51.929+0000 7f44307f8700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:56.792 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:56.793 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:56.793 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:56.835 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:56.836 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:57.097 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619950.865493_394.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:05:50.865493Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:05:57.098 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:05:57.098 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-48', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:57.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-48', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:57.099 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-48 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:57.141 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:57.141 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:57.339 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.39.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.63.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:05:57.340 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-48 2021-03-24T21:05:57.340 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-48', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:57.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-48', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:57.341 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-48 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:57.387 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:57.387 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:57.611 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"testobj-sse-c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.39.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"testobj-sse-kms","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.63.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:05:57.611 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-48 2021-03-24T21:05:57.714 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_encrypted_object_sync ... ok 2021-03-24T21:05:57.715 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-49 2021-03-24T21:05:57.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:57.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:57.936 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:57.975 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:57.975 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:58.142 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:05:58.173 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616619957.741943_398.1","last_update":"2021-03-24T21:05:57.741943Z"}] 2021-03-24T21:05:58.174 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:05:58.175 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:58.176 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:05:58.176 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:05:58.216 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:05:58.217 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:05:58.418 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619950.865493_394.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:05:50.865493Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:05:58.419 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616619957.741943_398.1 target=1_1616619950.865493_394.1 2021-03-24T21:05:58.938 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:05:58.933+0000 7f442eff5700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:58.939 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:05:58.933+0000 7f2d2bfff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:58.939 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:05:58.937+0000 7f442cff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:58.940 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:05:58.937+0000 7f442afed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:58.941 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:05:58.937+0000 7f2d217fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:58.941 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:05:58.937+0000 7f2d1afed700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:58.942 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:05:58.937+0000 7f2d297fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:58.943 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:05:58.937+0000 7f442fff7700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:58.944 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:05:58.937+0000 7f2d227fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:58.944 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:05:58.937+0000 7f2d1c7f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:58.945 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:05:58.937+0000 7f442a7ec700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:58.946 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:05:58.937+0000 7f442a7ec700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:58.946 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:05:58.937+0000 7f443a7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:58.946 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:05:58.937+0000 7f443a7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:58.947 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:05:58.937+0000 7f2d1cff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:05:58.947 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:05:58.937+0000 7f2d1cff1700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:06:03.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:03.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:03.421 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:03.463 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:03.463 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:03.671 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619957.741943_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:05:57.741943Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:03.672 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:06:03.672 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:03.673 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:03.673 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:03.674 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:03.754 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:03.755 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:04.013 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619957.741943_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:05:57.741943Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:04.014 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:04.015 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-49', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:04.015 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-49', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:04.016 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-49 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:04.059 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:04.059 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:04.254 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.38.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.66.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.45.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.50.5"}]} 2021-03-24T21:06:04.255 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-49 2021-03-24T21:06:04.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-49', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:04.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-49', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:04.256 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-49 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:04.300 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:04.300 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:04.506 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.38.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.66.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.50.5"}}}] 2021-03-24T21:06:04.506 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-49 2021-03-24T21:06:04.507 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-49', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:04.507 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-49', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:04.508 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-49 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:04.549 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:04.549 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:04.752 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.38.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.66.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.45.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.50.5"}]} 2021-03-24T21:06:04.753 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-49 2021-03-24T21:06:04.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-49', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:04.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-49', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:04.755 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-49 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:04.803 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:04.803 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:05.061 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.38.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.66.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.45.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.50.5"}}}] 2021-03-24T21:06:05.061 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-49 2021-03-24T21:06:05.062 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-49', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:05.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-49', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:05.063 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-49 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:05.104 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:05.105 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:05.303 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.53.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.74.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.52.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.68.5"}]} 2021-03-24T21:06:05.304 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-49 2021-03-24T21:06:05.304 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-49', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:05.305 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-49', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:05.306 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-49 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:05.352 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:05.353 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:05.548 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.53.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.74.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.68.5"}}}] 2021-03-24T21:06:05.675 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-49 2021-03-24T21:06:05.676 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-49 zones={test-zone2, test-zone1} 2021-03-24T21:06:05.687 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T21:06:05.736 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-24T21:06:05.737 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-24T21:06:05.737 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-24T21:06:05.738 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-24T21:06:05.738 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T21:06:05.744 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-24T21:06:05.744 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-24T21:06:05.744 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-24T21:06:05.745 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-24T21:06:05.799 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-24T21:06:05.852 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-24T21:06:05.965 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-24T21:06:06.068 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-24T21:06:06.123 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-24T21:06:06.230 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-24T21:06:06.332 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-24T21:06:06.387 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-24T21:06:06.497 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-24T21:06:06.608 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-24T21:06:06.665 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-24T21:06:06.777 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-24T21:06:06.884 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-49 zones={test-zone2, test-zone1} 2021-03-24T21:06:06.885 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:06:06.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:06:06.886 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:06:06.921 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:06.922 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:07.132 INFO:rgw_multi.tests:create bucket zone=test-zone1 name=gtvmdf-50 2021-03-24T21:06:07.355 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:07.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:07.356 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:07.397 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:07.563 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:07.592 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616619967.156478_402.1","last_update":"2021-03-24T21:06:07.156478Z"}] 2021-03-24T21:06:07.593 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:06:07.593 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:07.594 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:07.594 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:07.635 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:07.636 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:07.836 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619957.741943_398.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:05:57.741943Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:07.837 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616619967.156478_402.1 target=1_1616619957.741943_398.1 2021-03-24T21:06:08.345 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:06:08.341+0000 7f2d36ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:06:08.346 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:06:08.341+0000 7f443a7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:06:08.346 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:06:08.341+0000 7f442e7f4700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:06:08.347 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:06:08.341+0000 7f2d2b7fe700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:06:08.347 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:06:08.341+0000 7f2d28ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:06:08.347 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:06:08.341+0000 7f2d2bfff700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:06:08.347 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:06:08.341+0000 7f44317fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:06:08.348 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:06:08.341+0000 7f442dff3700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:06:08.348 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:06:08.345+0000 7f2d2a7fc700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:06:08.348 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:06:08.345+0000 7f2d22ffd700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:06:08.350 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:06:08.345+0000 7f44457fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:06:08.351 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:06:08.345+0000 7f44457fa700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:06:08.351 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:06:08.349+0000 7f2d1c7f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:06:08.352 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:06:08.345+0000 7f4430ff9700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:06:08.353 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:06:08.349+0000 7f2d1c7f0700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:06:08.353 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:06:08.349+0000 7f4445ffb700 -1 ERROR: _send_request(): get_sync_policy_handler() returned -2 2021-03-24T21:06:12.838 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:12.839 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:12.840 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:12.883 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:12.883 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:13.093 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619967.156478_402.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:07.156478Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:13.094 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:06:13.094 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:13.095 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:13.095 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:13.096 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:13.172 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:13.172 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:13.436 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619967.156478_402.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:07.156478Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:13.437 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:13.438 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:13.438 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:13.439 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-50 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:13.476 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:13.476 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:13.652 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.41.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.35.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.60.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.35.5"}]} 2021-03-24T21:06:13.652 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-50 2021-03-24T21:06:13.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-50', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:13.654 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-50', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:13.654 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-50 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:13.700 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:13.701 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:13.919 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.41.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.60.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.35.5"}}}] 2021-03-24T21:06:13.920 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone2 source_zone=test-zone1 bucket=gtvmdf-50 2021-03-24T21:06:13.920 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:13.921 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:13.921 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-50 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:13.961 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:13.962 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:14.162 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.41.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.35.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.60.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.35.5"}]} 2021-03-24T21:06:14.163 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-50 2021-03-24T21:06:14.163 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-50', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:14.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-50', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:14.165 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-50 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:14.208 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:14.208 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:14.474 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.41.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.35.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.60.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.35.5"}}}] 2021-03-24T21:06:14.474 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-50 2021-03-24T21:06:14.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-50', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:14.475 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bilog', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-50', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:14.476 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bilog status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-50 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:14.560 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:14.561 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:14.773 INFO:teuthology.orchestra.run.smithi063.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000001.64.5"},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":"00000000001.45.5"},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":"00000000001.52.5"},{"key":9,"val":""},{"key":10,"val":"00000000001.52.5"}]} 2021-03-24T21:06:14.774 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-50 2021-03-24T21:06:14.775 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:14.776 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone2', '--bucket', 'gtvmdf-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:14.777 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bucket sync markers --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone2 --bucket gtvmdf-50 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:14.823 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:14.823 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:15.034 INFO:teuthology.orchestra.run.smithi052.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"a","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.64.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"d","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.45.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"c","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"b","instance":"","ns":""},"count":1},"inc_marker":{"position":"00000000001.52.5"}}}] 2021-03-24T21:06:15.034 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone1 source_zone=test-zone2 bucket=gtvmdf-50 2021-03-24T21:06:15.035 INFO:tasks.rgw_multi.zone_rados:comparing bucket=gtvmdf-50 zones={test-zone2, test-zone1} 2021-03-24T21:06:15.044 DEBUG:tasks.rgw_multi.zone_rados:bucket1 objects: 2021-03-24T21:06:15.050 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-24T21:06:15.050 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-24T21:06:15.051 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-24T21:06:15.051 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-24T21:06:15.052 DEBUG:tasks.rgw_multi.zone_rados:bucket2 objects: 2021-03-24T21:06:15.099 DEBUG:tasks.rgw_multi.zone_rados:o=a 2021-03-24T21:06:15.099 DEBUG:tasks.rgw_multi.zone_rados:o=b 2021-03-24T21:06:15.100 DEBUG:tasks.rgw_multi.zone_rados:o=c 2021-03-24T21:06:15.101 DEBUG:tasks.rgw_multi.zone_rados:o=d 2021-03-24T21:06:15.155 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-24T21:06:15.209 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-24T21:06:15.313 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=a 2021-03-24T21:06:15.412 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-24T21:06:15.466 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-24T21:06:15.577 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=b 2021-03-24T21:06:15.675 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-24T21:06:15.735 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-24T21:06:15.850 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=c 2021-03-24T21:06:15.956 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-24T21:06:16.011 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-24T21:06:16.121 DEBUG:tasks.rgw_multi.zone_rados:comparing key name=d 2021-03-24T21:06:16.224 INFO:tasks.rgw_multi.zone_rados:success, bucket identical: bucket=gtvmdf-50 zones={test-zone2, test-zone1} 2021-03-24T21:06:16.224 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:06:16.225 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '1', '--rgw-sync-log-trim-min-cold-buckets', '0', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:06:16.226 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 1 --rgw-sync-log-trim-min-cold-buckets 0 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:06:16.263 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:16.264 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:16.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'gtvmdf-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:16.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'gtvmdf-50', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:16.489 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket gtvmdf-50 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:16.567 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:16.568 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:16.765 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2021-03-24T21:06:16.765 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'gtvmdf-49', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:16.766 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'gtvmdf-49', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:16.767 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket gtvmdf-49 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:16.844 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:16.845 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:17.053 INFO:teuthology.orchestra.run.smithi052.stdout:[{"op_id":"1#00000000001.38.5","op_tag":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4934.1923","op":"write","object":"a","instance":"","state":"complete","index_ver":1,"timestamp":"2021-03-24T21:05:57.749407947Z","ver":{"pool":9,"epoch":1},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-49:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4936.14"}]},{"op_id":"5#00000000001.66.5","op_tag":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4934.1929","op":"write","object":"d","instance":"","state":"complete","index_ver":1,"timestamp":"2021-03-24T21:05:57.894996885Z","ver":{"pool":9,"epoch":2},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-49:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4936.14"}]},{"op_id":"8#00000000001.45.5","op_tag":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4934.1927","op":"write","object":"c","instance":"","state":"complete","index_ver":1,"timestamp":"2021-03-24T21:05:57.846320451Z","ver":{"pool":9,"epoch":2},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-49:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4936.14"}]},{"op_id":"10#00000000001.50.5","op_tag":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4934.1925","op":"write","object":"b","instance":"","state":"complete","index_ver":1,"timestamp":"2021-03-24T21:05:57.798594087Z","ver":{"pool":9,"epoch":2},"bilog_flags":0,"versioned":false,"owner":"","owner_display_name":"","zones_trace":[{"entry":"662a64c9-f3f4-4dbc-9258-dc3b70a5b4df:gtvmdf-49:662a64c9-f3f4-4dbc-9258-dc3b70a5b4df.4936.14"}]}] 2021-03-24T21:06:17.054 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:06:17.055 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'autotrim', '--rgw-sync-log-trim-max-buckets', '999', '--rgw-sync-log-trim-min-cold-buckets', '999', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0'] 2021-03-24T21:06:17.055 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog autotrim --rgw-sync-log-trim-max-buckets 999 --rgw-sync-log-trim-min-cold-buckets 999 --cluster c1 --debug-rgw 0 --debug-ms 0 2021-03-24T21:06:17.132 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:17.133 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:18.225 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'list', '--bucket', 'gtvmdf-49', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:18.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'list', '--bucket', 'gtvmdf-49', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:18.226 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog list --bucket gtvmdf-49 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:18.306 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:18.312 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:18.513 INFO:teuthology.orchestra.run.smithi052.stdout:[] 2021-03-24T21:06:18.514 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_index_log_trim ... ok 2021-03-24T21:06:18.526 INFO:tasks.rgw_multisite_tests:rgw_multi.tests.test_bucket_creation_time ... ok 2021-03-24T21:06:18.527 INFO:tasks.rgw_multisite_tests:log information for manual testing ... ok 2021-03-24T21:06:18.528 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:18.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:18.536 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:18.595 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:18.595 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:18.762 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:18.795 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616619967.156478_402.1","last_update":"2021-03-24T21:06:07.156478Z"}] 2021-03-24T21:06:18.795 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:06:18.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:18.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:18.797 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:18.836 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:18.836 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:19.045 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619967.156478_402.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:07.156478Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:19.046 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:06:19.046 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:19.047 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:19.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:19.048 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:19.129 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:19.130 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:19.375 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619967.156478_402.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:07.156478Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:19.375 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:19.376 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:19.376 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:19.377 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:19.416 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:19.417 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:19.575 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:19.605 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616619967.156478_402.1","last_update":"2021-03-24T21:06:07.156478Z"}] 2021-03-24T21:06:19.606 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:19.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:19.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:19.607 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:19.647 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:19.648 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:19.906 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619967.156478_402.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:07.156478Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:19.907 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:19.936 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:19.937 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:19.937 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:19.973 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:19.974 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:20.147 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:20.179 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616619979.934673_412.1","last_update":"2021-03-24T21:06:19.934673Z"}] 2021-03-24T21:06:20.179 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:20.180 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:20.181 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:20.182 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:20.223 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:20.224 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:20.478 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619979.934673_412.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:19.934673Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:20.479 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:20.682 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:20.682 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:20.683 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:20.723 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:20.723 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:20.892 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:20.926 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616619979.934673_412.1","last_update":"2021-03-24T21:06:19.934673Z"}] 2021-03-24T21:06:20.926 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:20.927 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:20.927 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:20.928 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:20.970 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:20.970 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:21.228 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619979.934673_412.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:19.934673Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:21.228 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:21.300 INFO:tasks.rgw_multisite_tests:test low level implementation of s3 notifications ... ok 2021-03-24T21:06:21.302 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:21.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:21.304 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:21.342 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:21.343 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:21.497 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:21.528 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616619981.296686_414.1","last_update":"2021-03-24T21:06:21.296686Z"}] 2021-03-24T21:06:21.529 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:06:21.530 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:21.531 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:21.532 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:21.572 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:21.573 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:21.779 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619979.934673_412.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:19.934673Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:21.780 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616619981.296686_414.1 target=1_1616619979.934673_412.1 2021-03-24T21:06:26.781 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:26.782 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:26.783 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:26.823 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:26.823 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:27.024 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619981.296686_414.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:21.296686Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:27.025 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:06:27.025 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:27.026 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:27.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:27.027 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:27.108 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:27.109 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:27.357 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619981.296686_414.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:21.296686Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:27.358 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:27.358 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:27.359 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:27.360 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:27.402 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:27.402 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:27.571 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:27.601 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616619981.296686_414.1","last_update":"2021-03-24T21:06:21.296686Z"}] 2021-03-24T21:06:27.601 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:27.602 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:27.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:27.603 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:27.643 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:27.643 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:27.906 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619981.296686_414.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:21.296686Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:27.907 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:27.935 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:27.935 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:27.936 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:27.971 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:27.972 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:28.138 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:28.170 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616619987.933606_418.1","last_update":"2021-03-24T21:06:27.933606Z"}] 2021-03-24T21:06:28.170 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:28.171 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:28.171 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:28.172 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:28.213 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:28.213 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:28.466 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619987.923296_415.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:27.923296Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:28.467 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1616619987.933606_418.1 target=1_1616619987.923296_415.1 2021-03-24T21:06:33.468 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:33.469 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:33.469 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:33.511 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:33.512 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:33.872 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619987.933606_418.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:27.933606Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:33.873 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:33.911 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:33.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:33.912 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:33.952 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:33.953 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:34.120 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:34.150 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616619987.933606_418.1","last_update":"2021-03-24T21:06:27.933606Z"}] 2021-03-24T21:06:34.151 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:34.152 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:34.152 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:34.153 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:34.193 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:34.193 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:34.446 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616619987.933606_418.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:27.933606Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:34.447 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:34.867 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-52', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:34.868 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-52', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:34.868 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-52 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:34.909 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:34.909 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:35.110 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.51.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.33.5"},{"key":4,"val":"00000000001.45.5"},{"key":5,"val":"00000000001.47.5"},{"key":6,"val":""},{"key":7,"val":"00000000001.69.5"},{"key":8,"val":"00000000001.75.5"},{"key":9,"val":"00000000001.67.5"},{"key":10,"val":"00000000002.74.5"}]} 2021-03-24T21:06:35.111 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-52 2021-03-24T21:06:35.112 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-52', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:35.112 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-52', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:35.113 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-52 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:35.158 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:35.158 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:35.411 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:06:35.411 WARNING:rgw_multi.tests:bucket gtvmdf-52 zone test-zone3 behind zone test-zone1: shard=1 master=00000000002.51.5 target=, shard=3 master=00000000001.33.5 target=, shard=4 master=00000000001.45.5 target=, shard=5 master=00000000001.47.5 target=, shard=7 master=00000000001.69.5 target=, shard=8 master=00000000001.75.5 target=, shard=9 master=00000000001.67.5 target=, shard=10 master=00000000002.74.5 target= 2021-03-24T21:06:40.413 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-52', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:40.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-52', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:40.414 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-52 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:40.460 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:40.460 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:40.718 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.51.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.33.5"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.45.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.47.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.69.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.75.5"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.67.5"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.74.5"}}}] 2021-03-24T21:06:40.718 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-52 2021-03-24T21:06:40.734 WARNING:rgw_multi.tests_ps:superfluous records are found 2021-03-24T21:06:40.859 INFO:tasks.rgw_multisite_tests:test s3 records fetching ... ok 2021-03-24T21:06:40.861 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:40.861 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:40.862 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:40.903 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:40.903 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:41.076 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:41.106 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620000.856287_420.1","last_update":"2021-03-24T21:06:40.856287Z"}] 2021-03-24T21:06:41.107 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:06:41.108 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:41.108 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:41.109 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:41.150 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:41.151 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:41.334 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620000.856287_420.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:40.856287Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:41.334 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:06:41.335 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:41.335 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:41.336 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:41.336 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:41.416 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:41.417 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:41.678 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620000.856287_420.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:40.856287Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:41.679 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:41.679 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:41.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:41.681 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:41.720 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:41.720 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:41.889 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:41.921 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620000.856287_420.1","last_update":"2021-03-24T21:06:40.856287Z"}] 2021-03-24T21:06:41.921 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:41.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:41.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:41.923 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:41.964 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:41.964 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:42.219 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620000.856287_420.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:40.856287Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:42.220 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:42.248 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:42.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:42.250 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:42.287 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:42.287 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:42.451 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:42.483 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620002.247514_424.1","last_update":"2021-03-24T21:06:42.247514Z"}] 2021-03-24T21:06:42.483 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:42.484 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:42.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:42.485 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:42.527 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:42.527 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:42.784 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620002.247514_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:42.247514Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:42.784 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:42.853 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:42.854 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:42.854 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:42.896 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:42.896 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:43.071 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:43.105 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620002.247514_424.1","last_update":"2021-03-24T21:06:42.247514Z"}] 2021-03-24T21:06:43.106 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:43.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:43.107 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:43.108 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:43.147 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:43.148 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:43.404 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620002.247514_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:42.247514Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:43.404 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:43.487 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete ... ok 2021-03-24T21:06:43.488 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:43.488 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:43.489 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:43.525 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:43.525 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:43.692 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:43.723 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620003.485334_426.1","last_update":"2021-03-24T21:06:43.485334Z"}] 2021-03-24T21:06:43.724 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:06:43.725 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:43.725 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:43.726 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:43.768 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:43.769 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:43.990 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620002.247514_424.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:42.247514Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:43.991 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620003.485334_426.1 target=1_1616620002.247514_424.1 2021-03-24T21:06:48.992 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:48.993 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:48.994 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:49.037 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:49.037 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:49.224 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620003.485334_426.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:43.485334Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:49.225 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:06:49.225 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:49.226 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:49.226 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:49.227 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:49.304 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:49.304 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:49.560 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620003.485334_426.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:43.485334Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:49.561 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:49.561 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:49.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:49.562 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:49.604 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:49.605 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:49.755 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:49.781 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620003.485334_426.1","last_update":"2021-03-24T21:06:43.485334Z"}] 2021-03-24T21:06:49.782 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:49.783 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:49.783 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:49.784 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:49.826 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:49.826 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:50.088 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620003.485334_426.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:43.485334Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:50.088 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:50.298 INFO:tasks.rgw_multisite_tests:test s3 topics set/get/delete on master ... ok 2021-03-24T21:06:50.299 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:50.299 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:50.300 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:50.341 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:50.342 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:50.508 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:50.540 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620003.485334_426.1","last_update":"2021-03-24T21:06:43.485334Z"}] 2021-03-24T21:06:50.540 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:06:50.541 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:50.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:50.542 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:50.583 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:50.584 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:50.800 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620003.485334_426.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:43.485334Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:50.801 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:06:50.802 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:50.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:50.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:50.803 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:50.885 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:50.886 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:51.145 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620003.485334_426.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:43.485334Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:51.146 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:51.147 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:51.150 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:51.151 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:51.195 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:51.196 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:51.363 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:51.395 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620003.485334_426.1","last_update":"2021-03-24T21:06:43.485334Z"}] 2021-03-24T21:06:51.396 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:51.396 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:51.397 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:51.397 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:51.440 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:51.440 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:51.699 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620003.485334_426.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:43.485334Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:51.700 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:51.701 INFO:tasks.rgw_multisite_tests:test s3 topics with secret set/get/delete on master ... ok 2021-03-24T21:06:51.701 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:51.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:51.702 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:51.744 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:51.745 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:51.912 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:51.942 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620003.485334_426.1","last_update":"2021-03-24T21:06:43.485334Z"}] 2021-03-24T21:06:51.943 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:06:51.943 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:51.944 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:51.944 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:51.987 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:51.987 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:52.191 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620003.485334_426.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:43.485334Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:52.192 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:06:52.193 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:52.193 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:52.194 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:52.194 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:52.274 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:52.275 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:52.533 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620003.485334_426.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:43.485334Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:52.534 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:52.535 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:52.535 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:52.536 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:52.576 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:52.577 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:52.751 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:52.783 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620003.485334_426.1","last_update":"2021-03-24T21:06:43.485334Z"}] 2021-03-24T21:06:52.784 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:52.784 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:52.785 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:52.785 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:52.827 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:52.828 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:53.084 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620003.485334_426.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:43.485334Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:53.084 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:53.338 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2021-03-24T21:06:53.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests_ps.test_ps_s3_notification_filter_on_master ... ok 2021-03-24T21:06:53.339 INFO:tasks.rgw_multisite_tests:rgw_multi.tests_ps.test_ps_s3_notification_filter ... ok 2021-03-24T21:06:53.340 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:53.341 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:53.341 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:53.379 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:53.379 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:53.544 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:53.577 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620013.335725_432.1","last_update":"2021-03-24T21:06:53.335725Z"}] 2021-03-24T21:06:53.577 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:06:53.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:53.578 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:53.579 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:53.619 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:53.619 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:53.831 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620003.485334_426.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:43.485334Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:53.832 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620013.335725_432.1 target=1_1616620003.485334_426.1 2021-03-24T21:06:58.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:58.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:58.835 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:58.879 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:58.880 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:59.096 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620013.335725_432.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:53.335725Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:59.097 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:06:59.098 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:59.098 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:59.099 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:59.100 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:59.175 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:59.175 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:59.424 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620013.335725_432.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:53.335725Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:59.424 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:06:59.425 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:59.426 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:59.426 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:59.467 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:59.467 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:59.636 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:06:59.667 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620013.335725_432.1","last_update":"2021-03-24T21:06:53.335725Z"}] 2021-03-24T21:06:59.668 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:06:59.669 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:59.670 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:06:59.671 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:06:59.713 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:06:59.714 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:06:59.970 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620013.335725_432.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:53.335725Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:06:59.970 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:07:00.162 INFO:tasks.rgw_multisite_tests:test s3 notification set/get/delete on master ... ok 2021-03-24T21:07:00.163 INFO:tasks.rgw_multisite_tests:rgw_multi.tests_ps.test_objcet_timing ... ok 2021-03-24T21:07:00.163 INFO:tasks.rgw_multisite_tests:test pushing amqp s3 notification on master ... ok 2021-03-24T21:07:00.164 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2021-03-24T21:07:00.165 INFO:tasks.rgw_multisite_tests:test pushing kafka s3 notification on master ... ok 2021-03-24T21:07:00.165 INFO:tasks.rgw_multisite_tests:rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl ... ok 2021-03-24T21:07:00.166 INFO:tasks.rgw_multisite_tests:rgw_multi.tests_ps.test_ps_s3_notification_push_kafka_security_ssl_sasl ... ok 2021-03-24T21:07:00.166 INFO:tasks.rgw_multisite_tests:test deletion of multiple keys on master ... ok 2021-03-24T21:07:00.167 INFO:tasks.rgw_multisite_tests:test pushing http s3 notification on master ... ok 2021-03-24T21:07:00.168 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification ... ok 2021-03-24T21:07:00.169 INFO:tasks.rgw_multisite_tests:test that opaque id set in topic, is sent in notification on master ... ok 2021-03-24T21:07:00.169 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:00.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:00.171 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:00.210 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:00.211 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:00.378 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:07:00.410 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620020.159203_440.1","last_update":"2021-03-24T21:07:00.159203Z"}] 2021-03-24T21:07:00.410 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:07:00.411 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:00.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:00.412 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:00.454 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:00.455 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:00.676 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620013.335725_432.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:06:53.335725Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:00.677 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620020.159203_440.1 target=1_1616620013.335725_432.1 2021-03-24T21:07:05.679 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:05.680 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:05.681 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:05.722 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:05.723 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:05.926 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620020.159203_440.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:00.159203Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:05.926 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:07:05.927 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:07:05.928 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:05.928 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:05.929 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:06.008 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:06.008 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:06.265 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620020.159203_440.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:00.159203Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:06.265 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:07:06.266 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:06.266 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:06.267 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:06.307 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:06.308 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:06.474 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:07:06.498 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620020.159203_440.1","last_update":"2021-03-24T21:07:00.159203Z"}] 2021-03-24T21:07:06.499 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:07:06.500 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:06.500 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:06.501 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:06.540 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:06.540 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:06.802 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620020.159203_440.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:00.159203Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:06.803 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:07:06.822 INFO:tasks.rgw_multisite_tests:test set/get/delete of topic ... ok 2021-03-24T21:07:06.824 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:06.825 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:06.826 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:06.866 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:06.867 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:07.041 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:07:07.070 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620020.159203_440.1","last_update":"2021-03-24T21:07:00.159203Z"}] 2021-03-24T21:07:07.071 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:07:07.071 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:07.072 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:07.072 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:07.116 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:07.116 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:07.308 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620020.159203_440.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:00.159203Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:07.309 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:07:07.310 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:07:07.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:07.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:07.312 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:07.389 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:07.389 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:07.649 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620020.159203_440.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:00.159203Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:07.650 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:07:07.651 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:07.651 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:07.652 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:07.692 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:07.693 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:07.872 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:07:07.904 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620020.159203_440.1","last_update":"2021-03-24T21:07:00.159203Z"}] 2021-03-24T21:07:07.905 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:07:07.906 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:07.906 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:07.907 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:07.948 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:07.949 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:08.208 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620020.159203_440.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:00.159203Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:08.209 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:07:08.228 INFO:tasks.rgw_multisite_tests:test set topic with endpoint ... ok 2021-03-24T21:07:08.229 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:08.230 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:08.230 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:08.272 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:08.272 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:08.435 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:07:08.466 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620020.159203_440.1","last_update":"2021-03-24T21:07:00.159203Z"}] 2021-03-24T21:07:08.467 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:07:08.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:08.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:08.468 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:08.509 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:08.509 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:08.733 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620020.159203_440.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:00.159203Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:08.734 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:07:08.735 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:07:08.735 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:08.736 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:08.736 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:08.818 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:08.819 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:09.077 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620020.159203_440.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:00.159203Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:09.078 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:07:09.078 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:09.079 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:09.079 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:09.119 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:09.119 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:09.293 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:07:09.325 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620020.159203_440.1","last_update":"2021-03-24T21:07:00.159203Z"}] 2021-03-24T21:07:09.326 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:07:09.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:09.327 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:09.328 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:09.369 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:09.370 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:09.627 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620020.159203_440.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:00.159203Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:09.628 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:07:09.662 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:09.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:09.663 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:09.699 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:09.699 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:09.867 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:07:09.898 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620029.661175_444.1","last_update":"2021-03-24T21:07:09.661175Z"}] 2021-03-24T21:07:09.898 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:07:09.899 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:09.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:09.900 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:09.940 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:09.941 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:10.189 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620029.661175_444.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:09.661175Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:10.190 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:07:10.248 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification ... ok 2021-03-24T21:07:10.249 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:10.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:10.250 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:10.287 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:10.288 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:10.453 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:07:10.482 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620030.245862_446.1","last_update":"2021-03-24T21:07:10.245862Z"}] 2021-03-24T21:07:10.483 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:07:10.483 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:10.484 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:10.484 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:10.524 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:10.525 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:10.731 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620029.661175_444.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:09.661175Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:10.732 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620030.245862_446.1 target=1_1616620029.661175_444.1 2021-03-24T21:07:15.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:15.733 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:15.734 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:15.776 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:15.776 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:16.014 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620030.245862_446.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:10.245862Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:16.015 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:07:16.015 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:07:16.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:16.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:16.017 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:16.099 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:16.100 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:16.355 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620030.245862_446.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:10.245862Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:16.355 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:07:16.356 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:16.356 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:16.357 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:16.399 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:16.401 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:16.572 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:07:16.602 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620030.245862_446.1","last_update":"2021-03-24T21:07:10.245862Z"}] 2021-03-24T21:07:16.603 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:07:16.604 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:16.604 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:16.605 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:16.644 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:16.645 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:16.906 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620030.245862_446.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:10.245862Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:16.906 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:07:16.942 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:16.942 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:16.943 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:16.978 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:16.979 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:17.148 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:07:17.183 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620036.941142_450.1","last_update":"2021-03-24T21:07:16.941142Z"}] 2021-03-24T21:07:17.183 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:07:17.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:17.184 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:17.185 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:17.231 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:17.231 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:17.487 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620030.245862_446.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:10.245862Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:17.488 WARNING:rgw_multi.tests:zone test-zone3 behind master: shard=3 master=1_1616620036.941142_450.1 target=1_1616620030.245862_446.1 2021-03-24T21:07:22.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:22.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:22.491 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:22.533 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:22.534 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:22.793 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620036.941142_450.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:16.941142Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:22.794 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:07:22.856 INFO:tasks.rgw_multisite_tests:test set/get/delete of notification on specific events ... ok 2021-03-24T21:07:22.857 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:22.858 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:22.858 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:22.895 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:22.896 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:23.074 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:07:23.106 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620042.853529_452.1","last_update":"2021-03-24T21:07:22.853529Z"}] 2021-03-24T21:07:23.107 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:07:23.107 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:23.109 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:23.110 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:23.150 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:23.151 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:23.352 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620036.941142_450.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:16.941142Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:23.352 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620042.853529_452.1 target=1_1616620036.941142_450.1 2021-03-24T21:07:28.353 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:28.355 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:28.355 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:28.397 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:28.398 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:28.625 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620042.853529_452.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:22.853529Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:28.625 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:07:28.626 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:07:28.626 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:28.627 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:28.627 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:28.704 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:28.704 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:28.966 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620042.853529_452.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:22.853529Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:28.966 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:07:28.967 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:28.967 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:28.968 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:29.007 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:29.007 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:29.183 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:07:29.216 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620042.853529_452.1","last_update":"2021-03-24T21:07:22.853529Z"}] 2021-03-24T21:07:29.217 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:07:29.217 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:29.218 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:29.218 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:29.258 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:29.258 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:29.518 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620042.853529_452.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:22.853529Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:29.519 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:07:29.554 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:29.555 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:29.556 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:29.592 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:29.593 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:29.771 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:07:29.802 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620049.553685_456.1","last_update":"2021-03-24T21:07:29.553685Z"}] 2021-03-24T21:07:29.803 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:07:29.803 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:29.804 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:29.804 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:29.845 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:29.846 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:30.106 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620049.553685_456.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:29.553685Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:30.106 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:07:30.547 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:30.547 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:30.548 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:30.590 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:30.590 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:30.795 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.67.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.63.5"},{"key":4,"val":"00000000001.82.5"},{"key":5,"val":"00000000001.78.5"},{"key":6,"val":""},{"key":7,"val":"00000000001.75.5"},{"key":8,"val":"00000000001.56.5"},{"key":9,"val":"00000000001.82.5"},{"key":10,"val":"00000000002.61.5"}]} 2021-03-24T21:07:30.796 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-61 2021-03-24T21:07:30.796 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:30.797 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:30.798 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:30.843 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:30.843 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:31.100 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:07:31.101 WARNING:rgw_multi.tests:bucket gtvmdf-61 zone test-zone3 behind zone test-zone1: shard=1 master=00000000002.67.5 target=, shard=3 master=00000000001.63.5 target=, shard=4 master=00000000001.82.5 target=, shard=5 master=00000000001.78.5 target=, shard=7 master=00000000001.75.5 target=, shard=8 master=00000000001.56.5 target=, shard=9 master=00000000001.82.5 target=, shard=10 master=00000000002.61.5 target= 2021-03-24T21:07:36.104 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:36.105 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:36.105 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:36.151 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:36.151 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:36.410 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.67.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.63.5"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.82.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.78.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.75.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.56.5"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.82.5"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.5"}}}] 2021-03-24T21:07:36.411 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-61 2021-03-24T21:07:36.541 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:36.542 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-61', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:36.543 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-61 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:36.580 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:36.581 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:36.786 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.73.5"},{"key":2,"val":""},{"key":3,"val":"00000000002.69.5"},{"key":4,"val":"00000000002.84.5"},{"key":5,"val":"00000000002.80.5"},{"key":6,"val":""},{"key":7,"val":"00000000002.77.5"},{"key":8,"val":"00000000002.58.5"},{"key":9,"val":"00000000002.84.5"},{"key":10,"val":"00000000004.65.5"}]} 2021-03-24T21:07:36.787 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-61 2021-03-24T21:07:36.787 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:36.788 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:36.788 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:36.834 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:36.834 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:37.087 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.67.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.63.5"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.82.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.78.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.75.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.56.5"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.82.5"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.61.5"}}}] 2021-03-24T21:07:37.088 WARNING:rgw_multi.tests:bucket gtvmdf-61 zone test-zone3 behind zone test-zone1: shard=1 master=00000000004.73.5 target=00000000002.67.5, shard=3 master=00000000002.69.5 target=00000000001.63.5, shard=4 master=00000000002.84.5 target=00000000001.82.5, shard=5 master=00000000002.80.5 target=00000000001.78.5, shard=7 master=00000000002.77.5 target=00000000001.75.5, shard=8 master=00000000002.58.5 target=00000000001.56.5, shard=9 master=00000000002.84.5 target=00000000001.82.5, shard=10 master=00000000004.65.5 target=00000000002.61.5 2021-03-24T21:07:42.089 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:42.090 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-61', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:42.090 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-61 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:42.136 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:42.136 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:42.397 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.73.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.69.5"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.84.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.80.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.77.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.58.5"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.84.5"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.65.5"}}}] 2021-03-24T21:07:42.397 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-61 2021-03-24T21:07:42.451 INFO:tasks.rgw_multisite_tests:test set/get/delete of subscription ... ok 2021-03-24T21:07:42.452 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:42.453 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:42.453 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:42.491 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:42.491 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:42.668 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:07:42.700 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620062.449947_458.1","last_update":"2021-03-24T21:07:42.449947Z"}] 2021-03-24T21:07:42.701 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:07:42.702 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:42.702 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:42.703 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:42.745 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:42.745 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:42.956 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620049.553685_456.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:29.553685Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:42.956 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620062.449947_458.1 target=1_1616620049.553685_456.1 2021-03-24T21:07:47.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:47.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:47.959 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:47.999 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:48.000 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:48.220 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620062.449947_458.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:42.449947Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:48.221 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:07:48.221 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:07:48.222 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:48.222 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:48.223 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:48.300 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:48.300 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:48.554 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620062.449947_458.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:42.449947Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:48.554 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:07:48.555 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:48.556 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:48.556 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:48.595 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:48.595 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:48.779 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:07:48.809 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620062.449947_458.1","last_update":"2021-03-24T21:07:42.449947Z"}] 2021-03-24T21:07:48.810 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:07:48.810 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:48.811 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:48.811 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:48.851 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:48.852 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:49.110 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620062.449947_458.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:07:42.449947Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:07:49.111 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:07:49.600 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:49.601 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:49.602 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:49.643 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:49.643 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:49.857 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.60.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.77.5"},{"key":4,"val":"00000000001.47.5"},{"key":5,"val":"00000000001.55.5"},{"key":6,"val":""},{"key":7,"val":"00000000001.79.5"},{"key":8,"val":"00000000001.61.5"},{"key":9,"val":"00000000001.40.5"},{"key":10,"val":"00000000002.58.5"}]} 2021-03-24T21:07:49.858 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-62 2021-03-24T21:07:49.858 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:49.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:49.859 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:49.906 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:49.906 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:50.171 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:07:50.172 WARNING:rgw_multi.tests:bucket gtvmdf-62 zone test-zone3 behind zone test-zone1: shard=1 master=00000000002.60.5 target=, shard=3 master=00000000001.77.5 target=, shard=4 master=00000000001.47.5 target=, shard=5 master=00000000001.55.5 target=, shard=7 master=00000000001.79.5 target=, shard=8 master=00000000001.61.5 target=, shard=9 master=00000000001.40.5 target=, shard=10 master=00000000002.58.5 target= 2021-03-24T21:07:55.173 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:55.174 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:55.175 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:55.218 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:55.218 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:55.473 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.60.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.77.5"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.47.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.55.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.79.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.61.5"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.40.5"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.58.5"}}}] 2021-03-24T21:07:55.473 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-62 2021-03-24T21:07:55.919 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:55.919 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-62', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:55.920 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-62 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:55.960 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:55.961 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:56.169 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000001.57.5"},{"key":1,"val":"00000000003.62.5"},{"key":2,"val":""},{"key":3,"val":"00000000002.83.5"},{"key":4,"val":"00000000001.47.5"},{"key":5,"val":"00000000003.61.5"},{"key":6,"val":""},{"key":7,"val":"00000000003.85.5"},{"key":8,"val":"00000000002.63.5"},{"key":9,"val":"00000000002.42.5"},{"key":10,"val":"00000000003.60.5"}]} 2021-03-24T21:07:56.169 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-62 2021-03-24T21:07:56.170 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:56.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:07:56.171 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:07:56.220 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:07:56.220 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:07:56.474 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.60.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.77.5"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.47.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.55.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.79.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.61.5"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.40.5"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.58.5"}}}] 2021-03-24T21:07:56.474 WARNING:rgw_multi.tests:bucket gtvmdf-62 zone test-zone3 behind zone test-zone1: shard=0 master=00000000001.57.5 target=, shard=1 master=00000000003.62.5 target=00000000002.60.5, shard=3 master=00000000002.83.5 target=00000000001.77.5, shard=5 master=00000000003.61.5 target=00000000001.55.5, shard=7 master=00000000003.85.5 target=00000000001.79.5, shard=8 master=00000000002.63.5 target=00000000001.61.5, shard=9 master=00000000002.42.5 target=00000000001.40.5, shard=10 master=00000000003.60.5 target=00000000002.58.5 2021-03-24T21:08:01.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:01.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-62', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:01.477 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-62 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:01.524 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:01.525 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:01.789 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.57.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.62.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.83.5"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.47.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.61.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.85.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.63.5"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.42.5"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000003.60.5"}}}] 2021-03-24T21:08:01.790 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-62 2021-03-24T21:08:02.003 INFO:tasks.rgw_multisite_tests:test that events are only sent on incremental sync ... ok 2021-03-24T21:08:02.004 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:02.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:02.005 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:02.043 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:02.044 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:02.220 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:08:02.250 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620082.000689_473.1","last_update":"2021-03-24T21:08:02.000689Z"}] 2021-03-24T21:08:02.251 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:08:02.251 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:02.252 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:02.252 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:02.293 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:02.293 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:02.510 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620082.000689_473.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:02.000689Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:02.510 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:08:02.511 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:08:02.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:02.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:02.512 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:02.598 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:02.599 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:02.857 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620082.000689_473.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:02.000689Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:02.858 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:08:02.858 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:02.859 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:02.859 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:02.901 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:02.901 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:03.076 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:08:03.109 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620082.000689_473.1","last_update":"2021-03-24T21:08:02.000689Z"}] 2021-03-24T21:08:03.109 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:08:03.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:03.110 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:03.111 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:03.152 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:03.153 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:03.406 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620082.000689_473.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:02.000689Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:03.407 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:08:03.455 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:03.456 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:03.456 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:03.492 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:03.492 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:03.662 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:08:03.695 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620083.454333_477.1","last_update":"2021-03-24T21:08:03.454333Z"}] 2021-03-24T21:08:03.695 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:08:03.696 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:03.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:03.697 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:03.736 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:03.737 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:03.996 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620083.454333_477.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:03.454333Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:03.997 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:08:03.997 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:03.998 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:03.999 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:04.042 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:04.043 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:04.249 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":""},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":""},{"key":10,"val":""}]} 2021-03-24T21:08:04.250 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-63 2021-03-24T21:08:04.250 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:04.251 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:04.251 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:04.294 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:04.294 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:04.551 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:08:04.552 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-63 2021-03-24T21:08:05.027 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:05.027 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:05.028 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:05.067 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:05.068 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:05.268 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.86.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.52.5"},{"key":4,"val":"00000000001.68.5"},{"key":5,"val":"00000000001.72.5"},{"key":6,"val":""},{"key":7,"val":"00000000001.54.5"},{"key":8,"val":"00000000001.87.5"},{"key":9,"val":"00000000001.128.5"},{"key":10,"val":"00000000002.71.5"}]} 2021-03-24T21:08:05.269 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-63 2021-03-24T21:08:05.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:05.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:05.270 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:05.316 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:05.317 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:05.573 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.72.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.128.5"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:08:05.574 WARNING:rgw_multi.tests:bucket gtvmdf-63 zone test-zone3 behind zone test-zone1: shard=3 master=00000000001.52.5 target=, shard=4 master=00000000001.68.5 target=, shard=7 master=00000000001.54.5 target=, shard=8 master=00000000001.87.5 target=, shard=10 master=00000000002.71.5 target= 2021-03-24T21:08:10.576 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:10.577 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:10.577 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:10.622 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:10.623 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:10.881 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.52.5"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.68.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.72.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.54.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.87.5"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.128.5"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.5"}}}] 2021-03-24T21:08:10.882 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-63 2021-03-24T21:08:11.024 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:11.025 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-63', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:11.025 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-63 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:11.066 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:11.067 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:11.269 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000004.90.5"},{"key":2,"val":""},{"key":3,"val":"00000000002.54.5"},{"key":4,"val":"00000000002.70.5"},{"key":5,"val":"00000000002.74.5"},{"key":6,"val":""},{"key":7,"val":"00000000002.56.5"},{"key":8,"val":"00000000002.89.5"},{"key":9,"val":"00000000002.130.5"},{"key":10,"val":"00000000004.75.5"}]} 2021-03-24T21:08:11.270 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-63 2021-03-24T21:08:11.270 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:11.271 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:11.272 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:11.316 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:11.316 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:11.577 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.86.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.52.5"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.68.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.72.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.54.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.87.5"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.128.5"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.71.5"}}}] 2021-03-24T21:08:11.578 WARNING:rgw_multi.tests:bucket gtvmdf-63 zone test-zone3 behind zone test-zone1: shard=1 master=00000000004.90.5 target=00000000002.86.5, shard=3 master=00000000002.54.5 target=00000000001.52.5, shard=4 master=00000000002.70.5 target=00000000001.68.5, shard=5 master=00000000002.74.5 target=00000000001.72.5, shard=7 master=00000000002.56.5 target=00000000001.54.5, shard=8 master=00000000002.89.5 target=00000000001.87.5, shard=9 master=00000000002.130.5 target=00000000001.128.5, shard=10 master=00000000004.75.5 target=00000000002.71.5 2021-03-24T21:08:16.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:16.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-63', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:16.581 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-63 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:16.625 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:16.625 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:16.875 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.90.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.54.5"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.70.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.74.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.56.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.89.5"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.130.5"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000004.75.5"}}}] 2021-03-24T21:08:16.876 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-63 2021-03-24T21:08:16.989 INFO:tasks.rgw_multisite_tests:test subscriptions for different events ... ok 2021-03-24T21:08:16.990 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:16.991 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:16.992 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:17.029 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:17.030 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:17.205 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:08:17.237 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620096.987504_494.1","last_update":"2021-03-24T21:08:16.987504Z"}] 2021-03-24T21:08:17.238 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:08:17.239 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:17.239 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:17.240 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:17.279 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:17.279 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:17.492 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620083.454333_477.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:03.454333Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:17.492 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620096.987504_494.1 target=1_1616620083.454333_477.1 2021-03-24T21:08:22.494 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:22.495 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:22.496 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:22.537 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:22.538 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:22.752 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620096.987504_494.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:16.987504Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:22.752 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:08:22.753 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:08:22.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:22.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:22.755 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:22.834 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:22.834 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:23.091 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620096.987504_494.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:16.987504Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:23.091 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:08:23.092 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:23.092 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:23.093 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:23.133 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:23.134 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:23.311 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:08:23.342 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620096.987504_494.1","last_update":"2021-03-24T21:08:16.987504Z"}] 2021-03-24T21:08:23.342 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:08:23.343 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:23.343 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:23.344 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:23.386 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:23.387 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:23.649 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620096.987504_494.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:16.987504Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:23.650 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:08:23.685 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:23.686 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:23.687 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:23.723 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:23.723 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:23.889 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:08:23.920 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620103.684948_498.1","last_update":"2021-03-24T21:08:23.684948Z"}] 2021-03-24T21:08:23.921 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:08:23.921 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:23.922 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:23.923 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:23.963 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:23.964 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:24.217 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620103.684948_498.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:23.684948Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:24.218 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:08:28.816 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:28.816 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-64', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:28.817 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-64 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:28.859 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:28.860 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:29.066 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000010.89.5"},{"key":1,"val":"00000000010.100.5"},{"key":2,"val":"00000000009.64.5"},{"key":3,"val":"00000000010.114.5"},{"key":4,"val":"00000000007.91.5"},{"key":5,"val":"00000000007.92.5"},{"key":6,"val":"00000000008.87.5"},{"key":7,"val":"00000000010.83.5"},{"key":8,"val":"00000000010.97.5"},{"key":9,"val":"00000000010.111.5"},{"key":10,"val":"00000000009.106.5"}]} 2021-03-24T21:08:29.114 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-64 2021-03-24T21:08:29.115 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:29.115 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:29.116 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:29.161 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:29.162 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:29.416 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.85.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.100.5"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.60.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.114.5"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.91.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000006.90.5"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.87.5"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.83.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.97.5"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.107.5"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000009.106.5"}}}] 2021-03-24T21:08:29.473 WARNING:rgw_multi.tests:bucket gtvmdf-64 zone test-zone3 behind zone test-zone1: shard=0 master=00000000010.89.5 target=00000000008.85.5, shard=2 master=00000000009.64.5 target=00000000007.60.5, shard=5 master=00000000007.92.5 target=00000000006.90.5, shard=9 master=00000000010.111.5 target=00000000008.107.5 2021-03-24T21:08:34.475 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:34.476 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-64', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:34.476 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-64 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:34.522 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:34.522 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:34.782 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.89.5"}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.100.5"}}},{"key":2,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000009.64.5"}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.114.5"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.91.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000007.92.5"}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000008.87.5"}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.83.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.97.5"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000010.111.5"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000009.106.5"}}}] 2021-03-24T21:08:34.782 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-64 2021-03-24T21:08:35.592 INFO:tasks.rgw_multisite_tests:test incremental fetching of events from a subscription ... ok 2021-03-24T21:08:35.593 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:35.593 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:35.594 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:35.630 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:35.631 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:35.802 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:08:35.833 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620115.589727_534.1","last_update":"2021-03-24T21:08:35.589727Z"}] 2021-03-24T21:08:35.834 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:08:35.834 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:35.835 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:35.835 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:35.877 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:35.877 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:36.119 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620103.684948_498.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:23.684948Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:36.120 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620115.589727_534.1 target=1_1616620103.684948_498.1 2021-03-24T21:08:41.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:41.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:41.122 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:41.158 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:41.158 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:41.349 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620115.589727_534.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:35.589727Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:41.349 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:08:41.350 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:08:41.350 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:41.351 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:41.351 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:41.432 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:41.432 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:41.691 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620115.589727_534.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:35.589727Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:41.692 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:08:41.692 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:41.693 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:41.693 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:41.734 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:41.735 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:41.911 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:08:41.945 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620115.589727_534.1","last_update":"2021-03-24T21:08:35.589727Z"}] 2021-03-24T21:08:41.945 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:08:41.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:41.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:41.947 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:41.987 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:41.988 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:42.244 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620115.589727_534.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:35.589727Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:42.244 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:08:42.282 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:42.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:42.283 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:42.318 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:42.319 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:42.490 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:08:42.519 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620122.281004_545.1","last_update":"2021-03-24T21:08:42.281004Z"}] 2021-03-24T21:08:42.520 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:08:42.521 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:42.521 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:42.522 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:42.597 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:42.598 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:42.862 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620122.281004_545.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:42.281004Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:42.863 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:08:43.295 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:43.295 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-65', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:43.296 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-65 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:43.337 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:43.337 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:43.532 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":""},{"key":1,"val":"00000000002.108.5"},{"key":2,"val":""},{"key":3,"val":"00000000001.109.5"},{"key":4,"val":"00000000001.134.5"},{"key":5,"val":"00000000001.53.5"},{"key":6,"val":""},{"key":7,"val":"00000000001.92.5"},{"key":8,"val":"00000000001.98.5"},{"key":9,"val":"00000000001.155.5"},{"key":10,"val":"00000000002.114.5"}]} 2021-03-24T21:08:43.532 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-65 2021-03-24T21:08:43.533 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:43.533 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:43.534 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:43.578 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:43.579 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:43.840 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:08:43.841 WARNING:rgw_multi.tests:bucket gtvmdf-65 zone test-zone3 behind zone test-zone1: shard=1 master=00000000002.108.5 target=, shard=3 master=00000000001.109.5 target=, shard=4 master=00000000001.134.5 target=, shard=5 master=00000000001.53.5 target=, shard=7 master=00000000001.92.5 target=, shard=8 master=00000000001.98.5 target=, shard=9 master=00000000001.155.5 target=, shard=10 master=00000000002.114.5 target= 2021-03-24T21:08:48.843 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:48.849 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-65', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:48.850 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-65 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:48.896 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:48.897 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:49.157 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.108.5"}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.109.5"}}},{"key":4,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.134.5"}}},{"key":5,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.53.5"}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.92.5"}}},{"key":8,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.98.5"}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.155.5"}}},{"key":10,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000002.114.5"}}}] 2021-03-24T21:08:49.157 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-65 2021-03-24T21:08:49.332 INFO:tasks.rgw_multisite_tests:test acking of some events in a subscription ... ok 2021-03-24T21:08:49.334 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:49.334 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:49.335 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:49.373 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:49.373 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:49.541 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:08:49.573 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620129.329733_555.1","last_update":"2021-03-24T21:08:49.329733Z"}] 2021-03-24T21:08:49.574 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:08:49.574 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:49.575 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:49.575 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:49.616 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:49.616 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:49.824 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620122.281004_545.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:42.281004Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:49.825 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620129.329733_555.1 target=1_1616620122.281004_545.1 2021-03-24T21:08:54.827 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:54.831 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:54.832 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:54.913 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:54.914 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:55.129 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620129.329733_555.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:49.329733Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:55.130 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone2 2021-03-24T21:08:55.130 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:08:55.131 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:55.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:55.132 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:55.212 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:55.212 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:55.468 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620129.329733_555.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:49.329733Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:55.469 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:08:55.469 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:55.470 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:55.470 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:55.510 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:55.511 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:55.684 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:08:55.717 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620129.329733_555.1","last_update":"2021-03-24T21:08:49.329733Z"}] 2021-03-24T21:08:55.717 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:08:55.718 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:55.718 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:55.719 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:55.761 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:55.762 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:56.018 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620129.329733_555.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:49.329733Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:56.019 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:08:56.057 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:56.058 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:56.058 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:56.095 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:56.096 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:56.266 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:08:56.298 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620136.056496_562.1","last_update":"2021-03-24T21:08:56.056496Z"}] 2021-03-24T21:08:56.298 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone3 2021-03-24T21:08:56.299 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:56.300 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:56.300 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:56.342 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:56.342 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:56.702 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:08:56.702 INFO:rgw_multi.tests:finish meta checkpoint for zone=test-zone3 2021-03-24T21:08:57.011 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:57.011 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'bilog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--bucket', 'gtvmdf-66', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:57.012 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 bilog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --bucket gtvmdf-66 --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:57.055 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:57.055 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:57.235 INFO:tasks.rgw.c2.client.0.smithi063.stdout:*** Caught signal (Segmentation fault) ** 2021-03-24T21:08:57.235 INFO:tasks.rgw.c2.client.0.smithi063.stdout: in thread 7f2d297fa700 thread_name:rados_async 2021-03-24T21:08:57.238 INFO:tasks.rgw.c2.client.0.smithi063.stdout: ceph version 15.2.10-76-gdcf77bd9 (dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5) octopus (stable) 2021-03-24T21:08:57.238 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 1: (()+0x3f040) [0x7f2d61868040] 2021-03-24T21:08:57.239 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 2: (rgw::putobj::ETagVerifier_MPU::process(ceph::buffer::v15_2_0::list&&, unsigned long)+0x3f) [0x7f2d625c390f] 2021-03-24T21:08:57.239 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 3: (RGWRados::fetch_remote_obj(RGWObjectCtx&, rgw_user const&, req_info*, rgw_zone_id const&, rgw_obj const&, rgw_obj const&, RGWBucketInfo const&, RGWBucketInfo const*, std::optional, std::chrono::time_point > >*, std::chrono::time_point > >*, std::chrono::time_point > > const*, std::chrono::time_point > > const*, bool, char const*, char const*, RGWRados::AttrsMod, bool, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >&, RGWObjCategory, std::optional, std::chrono::time_point > >, std::__cxx11::basic_string, std::allocator >*, std::__cxx11::basic_string, std::allocator >*, void (*)(long, void*), void*, DoutPrefixProvider const*, RGWFetchObjFilter*, rgw_zone_set*, std::optional*)+0xd05) [0x7f2d62420785] 2021-03-24T21:08:57.239 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 4: (RGWAsyncFetchRemoteObj::_send_request()+0x3d6) [0x7f2d6236a8c6] 2021-03-24T21:08:57.240 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 5: (RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x20) [0x7f2d623646c0] 2021-03-24T21:08:57.240 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 6: (RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0xd) [0x7f2d6236cc0d] 2021-03-24T21:08:57.240 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 7: (ThreadPool::worker(ThreadPool::WorkThread*)+0x9fa) [0x7f2d581ff9ea] 2021-03-24T21:08:57.240 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 8: (ThreadPool::WorkThread::entry()+0x11) [0x7f2d582008d1] 2021-03-24T21:08:57.241 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 9: (()+0x76db) [0x7f2d571ba6db] 2021-03-24T21:08:57.241 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 10: (clone()+0x3f) [0x7f2d6194a71f] 2021-03-24T21:08:57.242 INFO:tasks.rgw.c2.client.0.smithi063.stdout:2021-03-24T21:08:57.235+0000 7f2d297fa700 -1 *** Caught signal (Segmentation fault) ** 2021-03-24T21:08:57.242 INFO:tasks.rgw.c2.client.0.smithi063.stdout: in thread 7f2d297fa700 thread_name:rados_async 2021-03-24T21:08:57.242 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 2021-03-24T21:08:57.243 INFO:tasks.rgw.c2.client.0.smithi063.stdout: ceph version 15.2.10-76-gdcf77bd9 (dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5) octopus (stable) 2021-03-24T21:08:57.243 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 1: (()+0x3f040) [0x7f2d61868040] 2021-03-24T21:08:57.243 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 2: (rgw::putobj::ETagVerifier_MPU::process(ceph::buffer::v15_2_0::list&&, unsigned long)+0x3f) [0x7f2d625c390f] 2021-03-24T21:08:57.243 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 3: (RGWRados::fetch_remote_obj(RGWObjectCtx&, rgw_user const&, req_info*, rgw_zone_id const&, rgw_obj const&, rgw_obj const&, RGWBucketInfo const&, RGWBucketInfo const*, std::optional, std::chrono::time_point > >*, std::chrono::time_point > >*, std::chrono::time_point > > const*, std::chrono::time_point > > const*, bool, char const*, char const*, RGWRados::AttrsMod, bool, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >&, RGWObjCategory, std::optional, std::chrono::time_point > >, std::__cxx11::basic_string, std::allocator >*, std::__cxx11::basic_string, std::allocator >*, void (*)(long, void*), void*, DoutPrefixProvider const*, RGWFetchObjFilter*, rgw_zone_set*, std::optional*)+0xd05) [0x7f2d62420785] 2021-03-24T21:08:57.244 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 4: (RGWAsyncFetchRemoteObj::_send_request()+0x3d6) [0x7f2d6236a8c6] 2021-03-24T21:08:57.244 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 5: (RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x20) [0x7f2d623646c0] 2021-03-24T21:08:57.244 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 6: (RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0xd) [0x7f2d6236cc0d] 2021-03-24T21:08:57.244 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 7: (ThreadPool::worker(ThreadPool::WorkThread*)+0x9fa) [0x7f2d581ff9ea] 2021-03-24T21:08:57.245 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 8: (ThreadPool::WorkThread::entry()+0x11) [0x7f2d582008d1] 2021-03-24T21:08:57.245 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 9: (()+0x76db) [0x7f2d571ba6db] 2021-03-24T21:08:57.245 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 10: (clone()+0x3f) [0x7f2d6194a71f] 2021-03-24T21:08:57.245 INFO:tasks.rgw.c2.client.0.smithi063.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-24T21:08:57.247 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 2021-03-24T21:08:57.248 INFO:teuthology.orchestra.run.smithi052.stdout:{"markers":[{"key":0,"val":"00000000001.61.5"},{"key":1,"val":""},{"key":2,"val":""},{"key":3,"val":""},{"key":4,"val":""},{"key":5,"val":""},{"key":6,"val":"00000000001.59.5"},{"key":7,"val":""},{"key":8,"val":""},{"key":9,"val":"00000000005.66.5"},{"key":10,"val":""}]} 2021-03-24T21:08:57.248 INFO:rgw_multi.tests:starting bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-66 2021-03-24T21:08:57.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-66', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:57.249 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-66', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:08:57.250 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-66 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:08:57.262 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 0> 2021-03-24T21:08:57.235+0000 7f2d297fa700 -1 *** Caught signal (Segmentation fault) ** 2021-03-24T21:08:57.263 INFO:tasks.rgw.c2.client.0.smithi063.stdout: in thread 7f2d297fa700 thread_name:rados_async 2021-03-24T21:08:57.263 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 2021-03-24T21:08:57.263 INFO:tasks.rgw.c2.client.0.smithi063.stdout: ceph version 15.2.10-76-gdcf77bd9 (dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5) octopus (stable) 2021-03-24T21:08:57.264 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 1: (()+0x3f040) [0x7f2d61868040] 2021-03-24T21:08:57.264 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 2: (rgw::putobj::ETagVerifier_MPU::process(ceph::buffer::v15_2_0::list&&, unsigned long)+0x3f) [0x7f2d625c390f] 2021-03-24T21:08:57.264 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 3: (RGWRados::fetch_remote_obj(RGWObjectCtx&, rgw_user const&, req_info*, rgw_zone_id const&, rgw_obj const&, rgw_obj const&, RGWBucketInfo const&, RGWBucketInfo const*, std::optional, std::chrono::time_point > >*, std::chrono::time_point > >*, std::chrono::time_point > > const*, std::chrono::time_point > > const*, bool, char const*, char const*, RGWRados::AttrsMod, bool, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >&, RGWObjCategory, std::optional, std::chrono::time_point > >, std::__cxx11::basic_string, std::allocator >*, std::__cxx11::basic_string, std::allocator >*, void (*)(long, void*), void*, DoutPrefixProvider const*, RGWFetchObjFilter*, rgw_zone_set*, std::optional*)+0xd05) [0x7f2d62420785] 2021-03-24T21:08:57.264 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 4: (RGWAsyncFetchRemoteObj::_send_request()+0x3d6) [0x7f2d6236a8c6] 2021-03-24T21:08:57.264 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 5: (RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x20) [0x7f2d623646c0] 2021-03-24T21:08:57.265 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 6: (RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0xd) [0x7f2d6236cc0d] 2021-03-24T21:08:57.265 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 7: (ThreadPool::worker(ThreadPool::WorkThread*)+0x9fa) [0x7f2d581ff9ea] 2021-03-24T21:08:57.265 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 8: (ThreadPool::WorkThread::entry()+0x11) [0x7f2d582008d1] 2021-03-24T21:08:57.265 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 9: (()+0x76db) [0x7f2d571ba6db] 2021-03-24T21:08:57.266 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 10: (clone()+0x3f) [0x7f2d6194a71f] 2021-03-24T21:08:57.266 INFO:tasks.rgw.c2.client.0.smithi063.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-24T21:08:57.266 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 2021-03-24T21:08:57.281 INFO:tasks.rgw.c2.client.0.smithi063.stdout: -314> 2021-03-24T21:08:57.235+0000 7f2d297fa700 -1 *** Caught signal (Segmentation fault) ** 2021-03-24T21:08:57.282 INFO:tasks.rgw.c2.client.0.smithi063.stdout: in thread 7f2d297fa700 thread_name:rados_async 2021-03-24T21:08:57.282 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 2021-03-24T21:08:57.283 INFO:tasks.rgw.c2.client.0.smithi063.stdout: ceph version 15.2.10-76-gdcf77bd9 (dcf77bd9ff89dfef682c4a2dda05ac5c8651dbe5) octopus (stable) 2021-03-24T21:08:57.283 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 1: (()+0x3f040) [0x7f2d61868040] 2021-03-24T21:08:57.283 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 2: (rgw::putobj::ETagVerifier_MPU::process(ceph::buffer::v15_2_0::list&&, unsigned long)+0x3f) [0x7f2d625c390f] 2021-03-24T21:08:57.284 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 3: (RGWRados::fetch_remote_obj(RGWObjectCtx&, rgw_user const&, req_info*, rgw_zone_id const&, rgw_obj const&, rgw_obj const&, RGWBucketInfo const&, RGWBucketInfo const*, std::optional, std::chrono::time_point > >*, std::chrono::time_point > >*, std::chrono::time_point > > const*, std::chrono::time_point > > const*, bool, char const*, char const*, RGWRados::AttrsMod, bool, std::map, std::allocator >, ceph::buffer::v15_2_0::list, std::less, std::allocator > >, std::allocator, std::allocator > const, ceph::buffer::v15_2_0::list> > >&, RGWObjCategory, std::optional, std::chrono::time_point > >, std::__cxx11::basic_string, std::allocator >*, std::__cxx11::basic_string, std::allocator >*, void (*)(long, void*), void*, DoutPrefixProvider const*, RGWFetchObjFilter*, rgw_zone_set*, std::optional*)+0xd05) [0x7f2d62420785] 2021-03-24T21:08:57.284 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 4: (RGWAsyncFetchRemoteObj::_send_request()+0x3d6) [0x7f2d6236a8c6] 2021-03-24T21:08:57.285 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 5: (RGWAsyncRadosProcessor::handle_request(RGWAsyncRadosRequest*)+0x20) [0x7f2d623646c0] 2021-03-24T21:08:57.285 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 6: (RGWAsyncRadosProcessor::RGWWQ::_process(RGWAsyncRadosRequest*, ThreadPool::TPHandle&)+0xd) [0x7f2d6236cc0d] 2021-03-24T21:08:57.285 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 7: (ThreadPool::worker(ThreadPool::WorkThread*)+0x9fa) [0x7f2d581ff9ea] 2021-03-24T21:08:57.286 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 8: (ThreadPool::WorkThread::entry()+0x11) [0x7f2d582008d1] 2021-03-24T21:08:57.286 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 9: (()+0x76db) [0x7f2d571ba6db] 2021-03-24T21:08:57.287 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 10: (clone()+0x3f) [0x7f2d6194a71f] 2021-03-24T21:08:57.287 INFO:tasks.rgw.c2.client.0.smithi063.stdout: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2021-03-24T21:08:57.288 INFO:tasks.rgw.c2.client.0.smithi063.stdout: 2021-03-24T21:08:57.297 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:08:57.297 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:08:57.336 INFO:tasks.rgw.c2.client.0.smithi063.stderr:daemon-helper: command crashed with signal 11 2021-03-24T21:08:57.549 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.66.5"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:08:57.550 WARNING:rgw_multi.tests:bucket gtvmdf-66 zone test-zone3 behind zone test-zone1: shard=0 master=00000000001.61.5 target=, shard=6 master=00000000001.59.5 target= 2021-03-24T21:08:58.876 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~0s 2021-03-24T21:09:02.551 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-66', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:02.552 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'bucket', 'sync', 'markers', '--rgw-zone', 'test-zone3', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--source-zone', 'test-zone1', '--bucket', 'gtvmdf-66', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:02.553 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 bucket sync markers --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --source-zone test-zone1 --bucket gtvmdf-66 --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:09:02.601 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:09:02.601 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:09:02.862 INFO:teuthology.orchestra.run.smithi063.stdout:[{"key":0,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.61.5"}}},{"key":1,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":2,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":3,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":4,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":5,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":6,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000001.59.5"}}},{"key":7,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":8,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}},{"key":9,"val":{"status":"incremental-sync","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":"00000000005.66.5"}}},{"key":10,"val":{"status":"init","full_marker":{"position":{"name":"","instance":"","ns":""},"count":0},"inc_marker":{"position":""}}}] 2021-03-24T21:09:02.863 INFO:rgw_multi.tests:finished bucket checkpoint for target_zone=test-zone3 source_zone=test-zone1 bucket=gtvmdf-66 2021-03-24T21:09:02.949 INFO:tasks.rgw_multisite_tests:test object creation notifications in using put/copy/post ... ok 2021-03-24T21:09:02.952 INFO:tasks.rgw_multisite_tests:test object creation s3 notifications in using put/copy/post on master ... ok 2021-03-24T21:09:02.953 INFO:tasks.rgw_multisite_tests:test multipart object upload on master ... ok 2021-03-24T21:09:02.954 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:02.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:02.955 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:09:02.995 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:09:02.996 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:09:03.166 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:09:03.198 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620142.947766_566.1","last_update":"2021-03-24T21:09:02.947766Z"}] 2021-03-24T21:09:03.199 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:09:03.199 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:03.200 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:03.201 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:09:03.245 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:09:03.245 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:09:03.470 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:09:03.471 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:09:04.484 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~6s 2021-03-24T21:09:08.472 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:08.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:08.475 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:09:08.515 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:09:08.516 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:09:08.740 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:09:08.740 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:09:10.090 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~11s 2021-03-24T21:09:13.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:13.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:13.744 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:09:13.785 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:09:13.786 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:09:14.013 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:09:14.013 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:09:15.696 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~17s 2021-03-24T21:09:19.015 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:19.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:19.016 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:09:19.056 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:09:19.057 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:09:19.260 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:09:19.261 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:09:21.302 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~22s 2021-03-24T21:09:24.262 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:24.263 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:24.264 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:09:24.308 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:09:24.309 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:09:24.535 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:09:24.535 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:09:26.911 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~28s 2021-03-24T21:09:29.537 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:29.538 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:29.538 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:09:29.580 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:09:29.581 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:09:29.800 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:09:29.800 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:09:32.518 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~34s 2021-03-24T21:09:34.802 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:34.803 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:34.803 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:09:34.843 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:09:34.844 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:09:35.068 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:09:35.069 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:09:38.124 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~39s 2021-03-24T21:09:40.070 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:40.071 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:40.071 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:09:40.113 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:09:40.113 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:09:40.325 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:09:40.326 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:09:43.731 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~45s 2021-03-24T21:09:45.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:45.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:45.328 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:09:45.371 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:09:45.372 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:09:45.592 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:09:45.593 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:09:49.337 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~50s 2021-03-24T21:09:50.595 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:50.596 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:50.597 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:09:50.641 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:09:50.641 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:09:50.868 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:09:50.869 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:09:54.943 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~56s 2021-03-24T21:09:55.870 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:55.871 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:09:55.872 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:09:55.912 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:09:55.912 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:09:56.135 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:09:56.136 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:10:00.598 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~62s 2021-03-24T21:10:01.137 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:01.137 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:01.138 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:10:01.182 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:10:01.183 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:10:01.392 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:10:01.393 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:10:06.204 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~67s 2021-03-24T21:10:06.394 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:06.395 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:06.396 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:10:06.436 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:10:06.436 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:10:06.659 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:10:06.660 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:10:11.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:11.662 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:11.663 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:10:11.705 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:10:11.706 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:10:11.811 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~73s 2021-03-24T21:10:11.927 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:10:11.928 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:10:16.929 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:16.930 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:16.931 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:10:16.972 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:10:16.973 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:10:17.166 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:10:17.166 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:10:17.420 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~79s 2021-03-24T21:10:22.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:22.169 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:22.170 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:10:22.216 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:10:22.217 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:10:22.417 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:10:22.418 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:10:23.025 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~84s 2021-03-24T21:10:27.420 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:27.421 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:27.422 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:10:27.463 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:10:27.463 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:10:27.672 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:10:27.673 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:10:28.633 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~90s 2021-03-24T21:10:32.674 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:32.675 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:32.676 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:10:32.716 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:10:32.716 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:10:32.920 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:10:32.921 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:10:34.240 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~95s 2021-03-24T21:10:37.922 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:37.923 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:37.924 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:10:37.967 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:10:37.967 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:10:38.177 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:10:38.178 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:10:39.846 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~101s 2021-03-24T21:10:43.179 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:43.180 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:43.181 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:10:43.225 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:10:43.226 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:10:43.438 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:10:43.438 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:10:45.451 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~107s 2021-03-24T21:10:48.440 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:48.444 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:48.444 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:10:48.486 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:10:48.487 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:10:48.696 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:10:48.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:10:51.058 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~112s 2021-03-24T21:10:53.698 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:53.699 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:53.699 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:10:53.741 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:10:53.741 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:10:53.949 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:10:53.950 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:10:56.665 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~118s 2021-03-24T21:10:58.952 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:58.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:10:58.954 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:10:58.995 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:10:58.996 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:10:59.192 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:10:59.193 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:11:02.272 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~123s 2021-03-24T21:11:04.194 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:04.195 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:04.195 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:11:04.233 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:11:04.234 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:11:04.433 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:11:04.434 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:11:07.879 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~129s 2021-03-24T21:11:09.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:09.436 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:09.437 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:11:09.478 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:11:09.479 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:11:09.696 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:11:09.697 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:11:13.485 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~135s 2021-03-24T21:11:14.699 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:14.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:14.755 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:11:14.795 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:11:14.796 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:11:15.005 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:11:15.006 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:11:19.093 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~140s 2021-03-24T21:11:20.007 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:20.008 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:20.008 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:11:20.048 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:11:20.048 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:11:20.250 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:11:20.251 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:11:24.699 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~146s 2021-03-24T21:11:25.253 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:25.253 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:25.254 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:11:25.295 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:11:25.296 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:11:25.503 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:11:25.504 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:11:30.305 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~151s 2021-03-24T21:11:30.505 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:30.505 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:30.506 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:11:30.547 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:11:30.547 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:11:30.755 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:11:30.755 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:11:35.756 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:35.757 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:35.758 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:11:35.799 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:11:35.800 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:11:35.912 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~157s 2021-03-24T21:11:36.014 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:11:36.015 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:11:41.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:41.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:41.017 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:11:41.059 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:11:41.059 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:11:41.266 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:11:41.266 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:11:41.517 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~163s 2021-03-24T21:11:46.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:46.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:46.269 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:11:46.309 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:11:46.309 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:11:46.503 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:11:46.504 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:11:47.124 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~168s 2021-03-24T21:11:51.506 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:51.506 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:51.507 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:11:51.549 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:11:51.550 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:11:51.742 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:11:51.743 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:11:52.730 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~174s 2021-03-24T21:11:56.744 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:56.745 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:11:56.745 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:11:56.786 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:11:56.787 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:11:56.993 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:11:56.994 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:11:58.336 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~179s 2021-03-24T21:12:01.995 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:01.996 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:01.997 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:12:02.040 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:12:02.041 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:12:02.246 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:12:02.247 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:12:03.942 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~185s 2021-03-24T21:12:07.249 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:07.250 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:07.251 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:12:07.291 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:12:07.292 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:12:07.487 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:12:07.488 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:12:09.549 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~191s 2021-03-24T21:12:12.489 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:12.490 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:12.491 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:12:12.533 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:12:12.533 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:12:12.741 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:12:12.742 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:12:15.155 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~196s 2021-03-24T21:12:17.743 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:17.744 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:17.745 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:12:17.785 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:12:17.785 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:12:17.991 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:12:17.992 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:12:20.762 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~202s 2021-03-24T21:12:22.993 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:22.994 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:22.995 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:12:23.035 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:12:23.036 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:12:23.260 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:12:23.260 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:12:26.368 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~207s 2021-03-24T21:12:28.261 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:28.262 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:28.263 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:12:28.303 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:12:28.303 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:12:28.508 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:12:28.509 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:12:31.975 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~213s 2021-03-24T21:12:33.510 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:33.511 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:33.512 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:12:33.551 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:12:33.552 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:12:33.759 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:12:33.759 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:12:37.581 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~219s 2021-03-24T21:12:38.761 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:38.761 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:38.762 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:12:38.803 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:12:38.803 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:12:39.014 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:12:39.014 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:12:43.188 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~224s 2021-03-24T21:12:44.016 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:44.017 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:44.017 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:12:44.059 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:12:44.060 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:12:44.265 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:12:44.266 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:12:48.793 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~230s 2021-03-24T21:12:49.267 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:49.268 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:49.269 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:12:49.309 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:12:49.309 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:12:49.510 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:12:49.511 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:12:54.399 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~236s 2021-03-24T21:12:54.511 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:54.512 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:54.513 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:12:54.553 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:12:54.553 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:12:54.762 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:12:54.763 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:12:59.764 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:59.765 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:12:59.766 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:12:59.808 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:12:59.808 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:13:00.005 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~241s 2021-03-24T21:13:00.064 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:13:00.065 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:13:05.067 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:05.068 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:05.069 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:13:05.116 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:13:05.117 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:13:05.303 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:13:05.304 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:13:05.667 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~247s 2021-03-24T21:13:10.306 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:10.307 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:10.308 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:13:10.349 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:13:10.349 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:13:10.559 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:13:10.560 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:13:11.273 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~252s 2021-03-24T21:13:15.561 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:15.562 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:15.563 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:13:15.603 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:13:15.603 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:13:15.802 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:13:15.803 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:13:16.880 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~258s 2021-03-24T21:13:20.804 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:20.805 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:20.805 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:13:20.848 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:13:20.848 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:13:21.060 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:13:21.060 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:13:22.486 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~264s 2021-03-24T21:13:26.061 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:26.063 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:26.064 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:13:26.107 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:13:26.109 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:13:26.314 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:13:26.315 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:13:28.091 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~269s 2021-03-24T21:13:31.316 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:31.317 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:31.318 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:13:31.359 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:13:31.359 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:13:31.576 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:13:31.577 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:13:33.697 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~275s 2021-03-24T21:13:36.578 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:36.579 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:36.580 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:13:36.621 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:13:36.621 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:13:36.835 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:13:36.836 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:13:39.303 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~280s 2021-03-24T21:13:41.837 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:41.838 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:41.839 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:13:41.880 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:13:41.880 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:13:42.099 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:13:42.100 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:13:44.909 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~286s 2021-03-24T21:13:47.101 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:47.102 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:47.103 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:13:47.148 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:13:47.148 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:13:47.325 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:13:47.326 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:13:50.516 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~292s 2021-03-24T21:13:52.327 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:52.328 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:52.329 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:13:52.371 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:13:52.371 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:13:52.583 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:13:52.584 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:13:56.122 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~297s 2021-03-24T21:13:57.585 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:57.586 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:13:57.587 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:13:57.629 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:13:57.629 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:13:57.831 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:13:57.832 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:14:01.727 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c2.rgw.client.0 is failed for ~303s 2021-03-24T21:14:01.728 INFO:tasks.daemonwatchdog.daemon_watchdog:BARK! unmounting mounts and killing all daemons 2021-03-24T21:14:01.728 ERROR:tasks.daemonwatchdog.daemon_watchdog:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2/qa/tasks/daemonwatchdog.py", line 42, in _run self.watch() File "/home/teuthworker/src/git.ceph.com_ceph-c_9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2/qa/tasks/daemonwatchdog.py", line 122, in watch self.bark() File "/home/teuthworker/src/git.ceph.com_ceph-c_9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2/qa/tasks/daemonwatchdog.py", line 58, in bark for mount in self.ctx.mounts.values(): File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/config.py", line 255, in __getattr__ raise AttributeError(name) AttributeError: mounts 2021-03-24T21:14:02.833 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:02.833 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:02.834 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:14:02.875 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:14:02.876 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:14:03.080 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:14:03.081 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:14:08.082 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:08.083 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:08.083 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:14:08.125 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:14:08.125 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:14:08.336 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:14:08.337 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:14:13.339 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:13.340 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:13.341 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:14:13.383 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:14:13.384 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:14:13.590 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:14:13.591 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:14:18.593 INFO:tasks.rgw_multisite_tests:test notification of deletion markers ... FAIL 2021-03-24T21:14:18.596 INFO:tasks.rgw_multisite_tests:test s3 notification of metadata on master ... ok 2021-03-24T21:14:18.597 INFO:tasks.rgw_multisite_tests:test s3 notification of tags on master ... ok 2021-03-24T21:14:18.597 INFO:tasks.rgw_multisite_tests:test s3 notification of object versions ... ok 2021-03-24T21:14:18.598 INFO:tasks.rgw_multisite_tests:test s3 notification of deletion markers on master ... ok 2021-03-24T21:14:18.599 INFO:tasks.rgw_multisite_tests:test pushing to http endpoint ... ok 2021-03-24T21:14:18.599 INFO:tasks.rgw_multisite_tests:test pushing to http endpoint s3 record format ... ok 2021-03-24T21:14:18.600 INFO:tasks.rgw_multisite_tests:test pushing to amqp endpoint ... ok 2021-03-24T21:14:18.600 INFO:tasks.rgw_multisite_tests:test pushing to amqp endpoint s3 record format ... ok 2021-03-24T21:14:18.601 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:18.602 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:18.603 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:14:18.643 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:14:18.644 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:14:18.825 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:14:18.855 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620142.947766_566.1","last_update":"2021-03-24T21:09:02.947766Z"}] 2021-03-24T21:14:18.856 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:14:18.857 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:18.857 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:18.858 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:14:18.900 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:14:18.900 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:14:19.123 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:14:19.124 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:14:24.125 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:24.126 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:24.127 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:14:24.168 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:14:24.168 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:14:24.368 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:14:24.369 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:14:29.370 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:29.371 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:29.372 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:14:29.415 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:14:29.415 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:14:29.618 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:14:29.619 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:14:34.621 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:34.621 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:34.622 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:14:34.664 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:14:34.665 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:14:34.883 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:14:34.883 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:14:39.884 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:39.885 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:39.886 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:14:39.928 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:14:39.929 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:14:40.136 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:14:40.136 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:14:45.138 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:45.139 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:45.139 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:14:45.180 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:14:45.180 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:14:45.384 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:14:45.385 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:14:50.386 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:50.387 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:50.388 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:14:50.430 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:14:50.431 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:14:50.650 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:14:50.651 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:14:55.653 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:55.653 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:14:55.654 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:14:55.696 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:14:55.696 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:14:55.907 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:14:55.907 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:15:00.908 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:00.909 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:00.910 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:15:00.951 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:15:00.951 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:15:01.165 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:15:01.165 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:15:06.167 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:06.168 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:06.168 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:15:06.210 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:15:06.211 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:15:06.410 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:15:06.411 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:15:11.412 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:11.413 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:11.413 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:15:11.456 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:15:11.456 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:15:11.659 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:15:11.660 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:15:16.661 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:16.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:16.663 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:15:16.704 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:15:16.704 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:15:16.912 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:15:16.913 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:15:21.914 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:21.915 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:21.916 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:15:21.956 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:15:21.957 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:15:22.182 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:15:22.183 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:15:27.184 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:27.185 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:27.185 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:15:27.227 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:15:27.227 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:15:27.433 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:15:27.434 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:15:32.436 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:32.437 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:32.438 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:15:32.480 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:15:32.481 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:15:32.695 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:15:32.695 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:15:37.697 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:37.698 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:37.698 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:15:37.738 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:15:37.739 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:15:37.955 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:15:37.955 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:15:42.957 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:42.958 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:42.959 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:15:42.999 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:15:43.000 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:15:43.211 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:15:43.211 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:15:48.213 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:48.214 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:48.215 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:15:48.257 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:15:48.258 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:15:48.466 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:15:48.466 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:15:53.467 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:53.468 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:53.469 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:15:53.510 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:15:53.510 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:15:53.740 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:15:53.741 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:15:58.742 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:58.743 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:15:58.744 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:15:58.786 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:15:58.787 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:15:59.012 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:15:59.012 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:16:04.014 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:04.016 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:04.016 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:16:04.056 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:16:04.056 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:16:04.285 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:16:04.285 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:16:09.287 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:09.288 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:09.288 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:16:09.326 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:16:09.327 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:16:09.544 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:16:09.544 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:16:14.546 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:14.551 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:14.551 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:16:14.593 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:16:14.594 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:16:14.815 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:16:14.816 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:16:19.817 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:19.818 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:19.818 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:16:19.861 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:16:19.861 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:16:20.078 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:16:20.079 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:16:25.080 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:25.081 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:25.082 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:16:25.127 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:16:25.127 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:16:25.321 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:16:25.322 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:16:30.323 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:30.324 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:30.325 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:16:30.363 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:16:30.364 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:16:30.578 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:16:30.578 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:16:35.579 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:35.580 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:35.581 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:16:35.622 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:16:35.623 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:16:35.847 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:16:35.847 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:16:40.849 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:40.850 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:40.850 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:16:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:16:40.893 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:16:41.128 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:16:41.129 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:16:46.130 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:46.131 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:46.132 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:16:46.173 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:16:46.173 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:16:46.367 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:16:46.368 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:16:51.369 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:51.370 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:51.371 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:16:51.412 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:16:51.412 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:16:51.627 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:16:51.627 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:16:56.628 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:56.629 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:16:56.630 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:16:56.672 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:16:56.672 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:16:56.896 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:16:56.897 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:17:01.898 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:01.899 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:01.899 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:17:01.940 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:17:01.940 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:17:02.161 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:17:02.162 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:17:07.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:07.164 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:07.165 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:17:07.205 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:17:07.403 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:17:07.408 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:17:07.408 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:17:12.410 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:12.411 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:12.412 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:17:12.452 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:17:12.452 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:17:12.675 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:17:12.675 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:17:17.677 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:17.677 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:17.678 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:17:17.721 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:17:17.721 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:17:17.951 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:17:17.952 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:17:22.954 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:22.954 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:22.955 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:17:22.997 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:17:22.997 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:17:23.207 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:17:23.208 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:17:28.209 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:28.210 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:28.211 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:17:28.250 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:17:28.250 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:17:28.470 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:17:28.470 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:17:33.471 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:33.473 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:33.474 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:17:33.515 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:17:33.515 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:17:33.731 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:17:33.732 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:17:38.733 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:38.734 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:38.734 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:17:38.776 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:17:38.777 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:17:39.002 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:17:39.003 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:17:44.004 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:44.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:44.005 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:17:44.046 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:17:44.047 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:17:44.255 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:17:44.256 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:17:49.258 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:49.258 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:49.259 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:17:49.299 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:17:49.300 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:17:49.518 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:17:49.519 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:17:54.520 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:54.522 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:54.523 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:17:54.566 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:17:54.567 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:17:54.792 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:17:54.793 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:17:59.795 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:59.796 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:17:59.796 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:17:59.838 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:17:59.839 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:18:00.044 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:18:00.045 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:18:05.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:05.053 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:05.054 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:18:05.097 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:18:05.097 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:18:05.304 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:18:05.304 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:18:10.305 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:10.306 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:10.307 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:18:10.349 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:18:10.350 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:18:10.570 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:18:10.571 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:18:15.573 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:15.573 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:15.574 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:18:15.616 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:18:15.616 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:18:15.841 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:18:15.842 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:18:20.844 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:20.844 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:20.845 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:18:20.888 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:18:20.889 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:18:21.100 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:18:21.101 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:18:26.102 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:26.103 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:26.103 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:18:26.143 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:18:26.144 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:18:26.354 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:18:26.355 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:18:31.356 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:31.357 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:31.358 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:18:31.399 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:18:31.399 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:18:31.604 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:18:31.605 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:18:36.606 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:36.607 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:36.608 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:18:36.651 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:18:36.652 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:18:36.869 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:18:36.870 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:18:41.871 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:41.872 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:41.873 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:18:41.913 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:18:41.913 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:18:42.122 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:18:42.123 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:18:47.124 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:47.125 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:47.125 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:18:47.166 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:18:47.167 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:18:47.376 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:18:47.376 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:18:52.379 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:52.380 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:52.381 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:18:52.426 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:18:52.426 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:18:52.648 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:18:52.648 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:18:57.650 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:57.650 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:18:57.651 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:18:57.691 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:18:57.692 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:18:57.909 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:18:57.909 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:19:02.911 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:02.912 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:02.912 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:19:02.952 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:19:02.952 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:19:03.167 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:19:03.168 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:19:08.169 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:08.170 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:08.170 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:19:08.212 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:19:08.213 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:19:08.426 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:19:08.426 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:19:13.427 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:13.428 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:13.429 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:19:13.472 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:19:13.472 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:19:13.693 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:19:13.694 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:19:18.695 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:18.696 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:18.697 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:19:18.738 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:19:18.739 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:19:18.956 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:19:18.957 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:19:23.958 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:23.959 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:23.960 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:19:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:19:24.001 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:19:24.218 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:19:24.219 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:19:29.220 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:29.221 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:29.221 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:19:29.260 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:19:29.261 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:19:29.475 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:19:29.476 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:19:34.479 INFO:tasks.rgw_multisite_tests:test notification status upon bucket deletion ... FAIL 2021-03-24T21:19:34.480 INFO:tasks.util.rgw:rgwadmin: c1.client.0 : ['mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:34.481 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c1', 'mdlog', 'status', '--rgw-zone', 'test-zone1', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c1', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:34.481 DEBUG:teuthology.orchestra.run.smithi052:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c1 mdlog status --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c1 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:19:34.524 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:19:34.525 INFO:teuthology.orchestra.run.smithi052.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:19:34.704 INFO:teuthology.orchestra.run.smithi052.stderr:No --period given, using current period=c083ad76-6371-41ed-840a-4187dd1c26c6 2021-03-24T21:19:34.736 INFO:teuthology.orchestra.run.smithi052.stdout:[{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"","last_update":"0.000000"},{"marker":"1_1616620142.947766_566.1","last_update":"2021-03-24T21:09:02.947766Z"}] 2021-03-24T21:19:34.736 INFO:rgw_multi.tests:starting meta checkpoint for zone=test-zone2 2021-03-24T21:19:34.737 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:34.738 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:34.738 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:19:34.780 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:19:34.781 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:19:35.001 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:19:35.002 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:19:40.003 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:40.004 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:40.005 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:19:40.045 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:19:40.046 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:19:40.261 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:19:40.262 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:19:45.263 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:45.264 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:45.265 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:19:45.308 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:19:45.308 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:19:45.517 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:19:45.517 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:19:50.518 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:50.519 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:50.520 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:19:50.561 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:19:50.562 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:19:50.769 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:19:50.769 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:19:55.771 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:55.772 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:19:55.772 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:19:55.812 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:19:55.812 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:19:56.030 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:19:56.031 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:20:01.032 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:01.033 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:01.034 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:20:01.075 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:20:01.075 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:20:01.267 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:20:01.268 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:20:06.269 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:06.270 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:06.270 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:20:06.316 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:20:06.316 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:20:06.523 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:20:06.524 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:20:11.525 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:11.526 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:11.526 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:20:11.567 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:20:11.567 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:20:11.777 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:20:11.778 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:20:16.779 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:16.780 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:16.781 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:20:16.821 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:20:16.822 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:20:17.028 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:20:17.028 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:20:22.030 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:22.030 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:22.031 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:20:22.072 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:20:22.072 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:20:22.280 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:20:22.281 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:20:27.282 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:27.283 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:27.283 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:20:27.323 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:20:27.323 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:20:27.537 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:20:27.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:20:32.539 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:32.540 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:32.540 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:20:32.581 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:20:32.582 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:20:32.789 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:20:32.790 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:20:37.791 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:37.792 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:37.793 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:20:37.833 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:20:37.833 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:20:38.044 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:20:38.045 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:20:43.046 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:43.047 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:43.048 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:20:43.088 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:20:43.088 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:20:43.287 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:20:43.288 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:20:48.289 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:48.290 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:48.290 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:20:48.333 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:20:48.333 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:20:48.538 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:20:48.538 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:20:53.540 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:53.541 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:53.541 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:20:53.584 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:20:53.585 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:20:53.797 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:20:53.797 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:20:58.799 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:58.800 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:20:58.800 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:20:58.841 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:20:58.841 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:20:59.052 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:20:59.053 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:21:04.055 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:04.056 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:04.056 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:21:04.098 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:21:04.098 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:21:04.308 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:21:04.309 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:21:09.310 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:09.311 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:09.311 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:21:09.355 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:21:09.355 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:21:09.588 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:21:09.588 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:21:14.589 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:14.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:14.610 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:21:14.651 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:21:14.651 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:21:14.862 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:21:14.862 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:21:19.864 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:19.865 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:19.866 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:21:19.908 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:21:19.908 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:21:20.118 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:21:20.119 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:21:25.121 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:25.122 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:25.122 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:21:25.165 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:21:25.165 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:21:25.364 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:21:25.365 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:21:30.367 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:30.367 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:30.368 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:21:30.410 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:21:30.410 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:21:30.621 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:21:30.621 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:21:35.622 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:35.624 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:35.624 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:21:35.669 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:21:35.669 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:21:35.879 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:21:35.880 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:21:40.881 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:40.882 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:40.883 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:21:40.923 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:21:40.924 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:21:41.139 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:21:41.140 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:21:46.141 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:46.142 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:46.143 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:21:46.185 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:21:46.186 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:21:46.387 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:21:46.387 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:21:51.388 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:51.389 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:51.390 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:21:51.431 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:21:51.431 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:21:51.639 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:21:51.640 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:21:56.642 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:56.642 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:21:56.643 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:21:56.686 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:21:56.687 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:21:56.902 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:21:56.903 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:22:01.904 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:01.905 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:01.906 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:22:01.949 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:22:02.129 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:22:02.160 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:22:02.161 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:22:07.164 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:07.165 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:07.166 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:22:07.207 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:22:07.207 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:22:07.405 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:22:07.406 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:22:12.407 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:12.408 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:12.409 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:22:12.452 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:22:12.452 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:22:12.662 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:22:12.663 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:22:17.666 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:17.667 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:17.667 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:22:17.711 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:22:17.711 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:22:17.932 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:22:17.933 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:22:22.935 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:22.936 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:22.936 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:22:22.979 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:22:22.979 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:22:23.186 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:22:23.187 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:22:28.188 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:28.189 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:28.190 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:22:28.233 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:22:28.234 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:22:28.432 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:22:28.433 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:22:33.434 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:33.434 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:33.435 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:22:33.477 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:22:33.478 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:22:33.692 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:22:33.693 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:22:38.694 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:38.695 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:38.696 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:22:38.736 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:22:38.736 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:22:38.945 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:22:38.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:22:43.947 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:43.948 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:43.949 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:22:43.990 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:22:43.991 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:22:44.206 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:22:44.206 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:22:49.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:49.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:49.209 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:22:49.248 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:22:49.248 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:22:49.470 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:22:49.471 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:22:54.473 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:54.474 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:54.475 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:22:54.515 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:22:54.516 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:22:54.751 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:22:54.751 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:22:59.753 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:59.754 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:22:59.755 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:22:59.797 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:22:59.797 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:23:00.016 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:23:00.017 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:23:05.018 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:05.019 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:05.020 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:23:05.064 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:23:05.065 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:23:05.253 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:23:05.253 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:23:10.255 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:10.256 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:10.257 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:23:10.299 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:23:10.300 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:23:10.522 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:23:10.523 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:23:15.524 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:15.525 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:15.526 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:23:15.567 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:23:15.567 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:23:15.786 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:23:15.786 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:23:20.788 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:20.789 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:20.790 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:23:20.836 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:23:20.836 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:23:21.055 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:23:21.055 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:23:26.056 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:26.057 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:26.058 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:23:26.100 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:23:26.101 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:23:26.326 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:23:26.327 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:23:31.329 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:31.330 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:31.330 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:23:31.371 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:23:31.371 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:23:31.605 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:23:31.606 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:23:36.607 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:36.609 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:36.610 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:23:36.653 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:23:36.654 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:23:36.880 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:23:36.881 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:23:41.882 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:41.883 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:41.884 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:23:41.925 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:23:41.925 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:23:42.145 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:23:42.145 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:23:47.147 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:47.148 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:47.148 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:23:47.187 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:23:47.187 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:23:47.398 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:23:47.398 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:23:52.400 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:52.401 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:52.401 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:23:52.440 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:23:52.441 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:23:52.660 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:23:52.660 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:23:57.662 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:57.663 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:23:57.664 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:23:57.705 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:23:57.706 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:23:57.944 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:23:57.945 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:24:02.946 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:02.947 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:02.948 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:24:02.988 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:24:02.988 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:24:03.206 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:24:03.207 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:24:08.208 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:08.209 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:08.210 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:24:08.252 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:24:08.253 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:24:08.483 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:24:08.484 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:24:13.486 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:13.487 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:13.487 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:24:13.527 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:24:13.528 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:24:13.760 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:24:13.760 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:24:18.761 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:18.762 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:18.763 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:24:18.805 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:24:18.805 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:24:19.023 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:24:19.024 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:24:24.025 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:24.026 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:24.027 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:24:24.067 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:24:24.068 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:24:24.300 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:24:24.301 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:24:29.303 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:29.303 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:29.304 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:24:29.344 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:24:29.344 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:24:29.563 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:24:29.564 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:24:34.565 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:34.566 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:34.567 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:24:34.607 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:24:34.607 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:24:34.835 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:24:34.835 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:24:39.836 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:39.837 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:39.838 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:24:39.880 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:24:39.881 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:24:40.108 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:24:40.109 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:24:45.110 INFO:tasks.util.rgw:rgwadmin: c2.client.0 : ['metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:45.111 DEBUG:tasks.util.rgw:rgwadmin: cmd=['adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'radosgw-admin', '--log-to-stderr', '--format', 'json', '-n', 'client.0', '--cluster', 'c2', 'metadata', 'sync', 'status', '--rgw-zone', 'test-zone2', '--rgw-zonegroup', 'test-zonegroup', '--rgw-realm', 'test-realm', '--cluster', 'c2', '--debug-rgw', '0', '--debug-ms', '0', '--rgw-cache-enabled', 'false'] 2021-03-24T21:24:45.112 DEBUG:teuthology.orchestra.run.smithi063:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage radosgw-admin --log-to-stderr --format json -n client.0 --cluster c2 metadata sync status --rgw-zone test-zone2 --rgw-zonegroup test-zonegroup --rgw-realm test-realm --cluster c2 --debug-rgw 0 --debug-ms 0 --rgw-cache-enabled false 2021-03-24T21:24:45.152 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setuser ceph since I am not root 2021-03-24T21:24:45.152 INFO:teuthology.orchestra.run.smithi063.stderr:ignoring --setgroup ceph since I am not root 2021-03-24T21:24:45.380 INFO:teuthology.orchestra.run.smithi063.stdout:{"sync_status":{"info":{"status":"sync","num_shards":4,"period":"c083ad76-6371-41ed-840a-4187dd1c26c6","realm_epoch":4},"markers":[{"key":0,"val":{"state":1,"marker":"1_1616618820.151371_27.1","next_step_marker":"","total_entries":1,"pos":0,"timestamp":"2021-03-24T20:47:00.151371Z","realm_epoch":2}},{"key":1,"val":{"state":1,"marker":"","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"0.000000","realm_epoch":2}},{"key":2,"val":{"state":1,"marker":"1_1616618986.101275_85.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T20:49:46.101275Z","realm_epoch":2}},{"key":3,"val":{"state":1,"marker":"1_1616620136.056496_562.1","next_step_marker":"","total_entries":0,"pos":0,"timestamp":"2021-03-24T21:08:56.056496Z","realm_epoch":4}}]},"full_sync":{"total":1,"complete":1}} 2021-03-24T21:24:45.380 WARNING:rgw_multi.tests:zone test-zone2 behind master: shard=3 master=1_1616620142.947766_566.1 target=1_1616620136.056496_562.1 2021-03-24T21:24:50.382 INFO:tasks.rgw_multisite_tests:test creating a subscription when no topic info exists ... FAIL 2021-03-24T21:24:50.385 INFO:tasks.rgw_multisite_tests:test updating topic associated with a notification ... ok 2021-03-24T21:24:50.386 INFO:tasks.rgw_multisite_tests:test updating the topic of a notification ... ok 2021-03-24T21:24:50.386 INFO:tasks.rgw_multisite_tests:test notification creation with multiple topics ... ok 2021-03-24T21:24:50.387 INFO:tasks.rgw_multisite_tests: 2021-03-24T21:24:50.388 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-03-24T21:24:50.389 INFO:tasks.rgw_multisite_tests:FAIL: test notification of deletion markers 2021-03-24T21:24:50.389 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-03-24T21:24:50.389 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-03-24T21:24:50.390 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-03-24T21:24:50.390 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-03-24T21:24:50.390 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 2764, in test_ps_versioned_deletion 2021-03-24T21:24:50.391 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-03-24T21:24:50.391 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 573, in init_env 2021-03-24T21:24:50.391 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-03-24T21:24:50.392 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 199, in zonegroup_meta_checkpoint 2021-03-24T21:24:50.392 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-03-24T21:24:50.392 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 188, in zone_meta_checkpoint 2021-03-24T21:24:50.393 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-03-24T21:24:50.393 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2021-03-24T21:24:50.393 INFO:tasks.rgw_multisite_tests: 2021-03-24T21:24:50.394 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-03-24T21:24:50.394 INFO:tasks.rgw_multisite_tests:FAIL: test notification status upon bucket deletion 2021-03-24T21:24:50.395 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-03-24T21:24:50.396 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-03-24T21:24:50.396 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-03-24T21:24:50.397 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-03-24T21:24:50.397 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 3470, in test_ps_delete_bucket 2021-03-24T21:24:50.397 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-03-24T21:24:50.398 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 573, in init_env 2021-03-24T21:24:50.398 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-03-24T21:24:50.398 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 199, in zonegroup_meta_checkpoint 2021-03-24T21:24:50.399 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-03-24T21:24:50.399 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 188, in zone_meta_checkpoint 2021-03-24T21:24:50.400 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-03-24T21:24:50.400 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2021-03-24T21:24:50.400 INFO:tasks.rgw_multisite_tests: 2021-03-24T21:24:50.401 INFO:tasks.rgw_multisite_tests:====================================================================== 2021-03-24T21:24:50.401 INFO:tasks.rgw_multisite_tests:FAIL: test creating a subscription when no topic info exists 2021-03-24T21:24:50.402 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-03-24T21:24:50.402 INFO:tasks.rgw_multisite_tests:Traceback (most recent call last): 2021-03-24T21:24:50.402 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/virtualenv/lib/python3.6/site-packages/nose/case.py", line 198, in runTest 2021-03-24T21:24:50.403 INFO:tasks.rgw_multisite_tests: self.test(*self.arg) 2021-03-24T21:24:50.403 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 3539, in test_ps_missing_topic 2021-03-24T21:24:50.403 INFO:tasks.rgw_multisite_tests: master_zone, ps_zone = init_env() 2021-03-24T21:24:50.404 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests_ps.py", line 573, in init_env 2021-03-24T21:24:50.404 INFO:tasks.rgw_multisite_tests: zonegroup_meta_checkpoint(zonegroup) 2021-03-24T21:24:50.404 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 199, in zonegroup_meta_checkpoint 2021-03-24T21:24:50.405 INFO:tasks.rgw_multisite_tests: zone_meta_checkpoint(zone, meta_master_zone, master_status) 2021-03-24T21:24:50.405 INFO:tasks.rgw_multisite_tests: File "/home/teuthworker/src/github.com_ceph_ceph_octopus/src/test/rgw/rgw_multi/tests.py", line 188, in zone_meta_checkpoint 2021-03-24T21:24:50.405 INFO:tasks.rgw_multisite_tests: assert False, 'failed meta checkpoint for zone=%s' % zone.name 2021-03-24T21:24:50.406 INFO:tasks.rgw_multisite_tests:AssertionError: failed meta checkpoint for zone=test-zone2 2021-03-24T21:24:50.406 INFO:tasks.rgw_multisite_tests: 2021-03-24T21:24:50.407 INFO:tasks.rgw_multisite_tests:---------------------------------------------------------------------- 2021-03-24T21:24:50.407 INFO:tasks.rgw_multisite_tests:Ran 74 tests in 2153.054s 2021-03-24T21:24:50.407 INFO:tasks.rgw_multisite_tests: 2021-03-24T21:24:50.408 INFO:tasks.rgw_multisite_tests:FAILED (failures=3) 2021-03-24T21:24:50.408 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2/qa/tasks/rgw_multisite_tests.py", line 93, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2021-03-24T21:24:50.555 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=be00aec9dac74943a22fb926c2c98414 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/run_tasks.py", line 94, in run_tasks manager.__enter__() File "/home/teuthworker/src/git.ceph.com_git_teuthology_6b3150e9e0aa7ca432e26f31d87920ebd77f3708/teuthology/task/__init__.py", line 123, in __enter__ self.begin() File "/home/teuthworker/src/git.ceph.com_ceph-c_9e70f1c279e3b59cde5f6db62ba3c9527a0ce6f2/qa/tasks/rgw_multisite_tests.py", line 93, in begin assert nose.run(argv=argv, config=conf), 'rgw multisite test failures' AssertionError: rgw multisite test failures 2021-03-24T21:24:50.557 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite-tests 2021-03-24T21:24:50.565 DEBUG:teuthology.run_tasks:Unwinding manager rgw-multisite 2021-03-24T21:24:50.600 DEBUG:teuthology.run_tasks:Unwinding manager rgw 2021-03-24T21:24:50.633 DEBUG:tasks.rgw.c1.client.0:waiting for process to exit 2021-03-24T21:24:50.633 INFO:teuthology.orchestra.run:waiting for 300 2021-03-24T21:24:50.638 INFO:tasks.rgw.c1.client.0.smithi052.stdout:2021-03-24T21:24:50.631+0000 7f319e7fc700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8000 -n client.0 --cluster c1 -k /etc/ceph/c1.client.0.keyring --log-file /var/log/ceph/rgw.c1.client.0.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock --foreground --rgw-zone test-zone1 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 15074) UID: 0 2021-03-24T21:24:50.638 INFO:tasks.rgw.c1.client.0.smithi052.stdout:2021-03-24T21:24:50.631+0000 7f31ba3cc0c0 -1 shutting down 2021-03-24T21:24:52.032 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 is failed for ~0s 2021-03-24T21:24:56.736 INFO:tasks.rgw.c1.client.0:Stopped 2021-03-24T21:24:56.737 DEBUG:teuthology.orchestra.run.smithi052:> rm -f /home/ubuntu/cephtest/rgw.opslog.c1.client.0.sock 2021-03-24T21:24:56.744 DEBUG:teuthology.orchestra.run.smithi052:> rm -f /home/ubuntu/cephtest/vault-token 2021-03-24T21:24:56.792 DEBUG:tasks.rgw.c2.client.0:waiting for process to exit 2021-03-24T21:24:56.794 INFO:teuthology.orchestra.run:waiting for 300 2021-03-24T21:24:56.794 INFO:tasks.rgw.c2.client.0:Stopped 2021-03-24T21:24:56.795 DEBUG:teuthology.orchestra.run.smithi063:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.0.sock 2021-03-24T21:24:56.802 DEBUG:teuthology.orchestra.run.smithi063:> rm -f /home/ubuntu/cephtest/vault-token 2021-03-24T21:24:56.851 DEBUG:tasks.rgw.c2.client.1:waiting for process to exit 2021-03-24T21:24:56.852 INFO:teuthology.orchestra.run:waiting for 300 2021-03-24T21:24:56.894 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:24:56.895+0000 7f44567fc700 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term radosgw --rgw-frontends beast port=8001 -n client.1 --cluster c2 -k /etc/ceph/c2.client.1.keyring --log-file /var/log/ceph/rgw.c2.client.1.log --rgw_ops_log_socket_path /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock --foreground --rgw-zone test-zone3 --rgw-zonegroup test-zonegroup --rgw-realm test-realm (PID: 29498) UID: 0 2021-03-24T21:24:56.895 INFO:tasks.rgw.c2.client.1.smithi063.stdout:2021-03-24T21:24:56.895+0000 7f44723a50c0 -1 shutting down 2021-03-24T21:24:57.544 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon c1.rgw.client.0 has been restored 2021-03-24T21:25:02.953 INFO:tasks.rgw.c2.client.1:Stopped 2021-03-24T21:25:02.954 DEBUG:teuthology.orchestra.run.smithi063:> rm -f /home/ubuntu/cephtest/rgw.opslog.c2.client.1.sock 2021-03-24T21:25:02.961 DEBUG:teuthology.orchestra.run.smithi063:> rm -f /home/ubuntu/cephtest/vault-token 2021-03-24T21:25:03.012 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2021-03-24T21:25:03.119 INFO:tasks.ceph.ceph_manager.c2:Canceling any pending splits or merges... 2021-03-24T21:25:03.154 DEBUG:teuthology.orchestra.run.smithi063:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c2 osd dump --format=json 2021-03-24T21:25:03.439 INFO:teuthology.orchestra.run.smithi063.stdout: 2021-03-24T21:25:03.440 INFO:teuthology.orchestra.run.smithi063.stdout:{"epoch":66,"fsid":"19ff7635-a2a7-4347-a6a0-ab464f77d5fb","created":"2021-03-24T20:45:10.980569+0000","modified":"2021-03-24T21:00:06.263021+0000","last_up_change":"2021-03-24T20:45:16.069521+0000","last_in_change":"2021-03-24T20:45:16.069521+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-24T20:45:15.056543+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-24T20:45:19.412470+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"14","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2021-03-24T20:45:35.916346+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"19","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2021-03-24T20:45:37.412766+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2021-03-24T20:45:50.788857+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2021-03-24T20:45:52.242998+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2021-03-24T20:45:54.219899+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"27","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2021-03-24T20:45:56.268375+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"30","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"test-zone2.rgw.buckets.data","create_time":"2021-03-24T20:46:21.724370+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"test-zone2.rgw.log","create_time":"2021-03-24T20:46:23.353825+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":11,"pool_name":"test-zone2.rgw.control","create_time":"2021-03-24T20:46:25.098161+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"37","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":12,"pool_name":"test-zone2.rgw.meta","create_time":"2021-03-24T20:46:27.156764+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"40","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":13,"pool_name":"test-zone3.rgw.buckets.data","create_time":"2021-03-24T20:46:40.047827+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"43","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":14,"pool_name":"test-zone3.rgw.log","create_time":"2021-03-24T20:46:41.393133+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":15,"pool_name":"test-zone3.rgw.control","create_time":"2021-03-24T20:46:43.214424+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"47","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":16,"pool_name":"test-zone3.rgw.meta","create_time":"2021-03-24T20:46:45.289109+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"50","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":17,"pool_name":"test-zone2.rgw.buckets.index","create_time":"2021-03-24T20:49:00.642086+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"54","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":18,"pool_name":"test-zone3.rgw.buckets.index","create_time":"2021-03-24T20:49:01.638636+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"55","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":19,"pool_name":"test-zone2.rgw.otp","create_time":"2021-03-24T20:52:25.945165+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"58","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":20,"pool_name":"remove.rgw.log","create_time":"2021-03-24T20:59:58.949978+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":21,"pool_name":"remove.rgw.control","create_time":"2021-03-24T21:00:00.320454+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"63","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":22,"pool_name":"remove.rgw.meta","create_time":"2021-03-24T21:00:02.381514+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"66","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"4716e8a9-b4a7-4c50-a4e7-8c860600febd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6816","nonce":12758},{"type":"v1","addr":"172.21.15.63:6817","nonce":12758}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6818","nonce":12758},{"type":"v1","addr":"172.21.15.63:6819","nonce":12758}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6822","nonce":12758},{"type":"v1","addr":"172.21.15.63:6823","nonce":12758}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6820","nonce":12758},{"type":"v1","addr":"172.21.15.63:6821","nonce":12758}]},"public_addr":"172.21.15.63:6817/12758","cluster_addr":"172.21.15.63:6819/12758","heartbeat_back_addr":"172.21.15.63:6823/12758","heartbeat_front_addr":"172.21.15.63:6821/12758","state":["exists","up"]},{"osd":1,"uuid":"e72ecbde-5057-4f63-b5ce-ce1fe8f0cfa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6808","nonce":12759},{"type":"v1","addr":"172.21.15.63:6809","nonce":12759}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6810","nonce":12759},{"type":"v1","addr":"172.21.15.63:6811","nonce":12759}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6814","nonce":12759},{"type":"v1","addr":"172.21.15.63:6815","nonce":12759}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6812","nonce":12759},{"type":"v1","addr":"172.21.15.63:6813","nonce":12759}]},"public_addr":"172.21.15.63:6809/12759","cluster_addr":"172.21.15.63:6811/12759","heartbeat_back_addr":"172.21.15.63:6815/12759","heartbeat_front_addr":"172.21.15.63:6813/12759","state":["exists","up"]},{"osd":2,"uuid":"d87e26ae-d46d-4786-b961-8369e589f488","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":63,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6800","nonce":12757},{"type":"v1","addr":"172.21.15.63:6801","nonce":12757}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6802","nonce":12757},{"type":"v1","addr":"172.21.15.63:6803","nonce":12757}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6806","nonce":12757},{"type":"v1","addr":"172.21.15.63:6807","nonce":12757}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.63:6804","nonce":12757},{"type":"v1","addr":"172.21.15.63:6805","nonce":12757}]},"public_addr":"172.21.15.63:6801/12757","cluster_addr":"172.21.15.63:6803/12757","heartbeat_back_addr":"172.21.15.63:6807/12757","heartbeat_front_addr":"172.21.15.63:6805/12757","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:45:14.259698+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:45:13.993033+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:45:13.945932+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-24T21:25:03.454 DEBUG:teuthology.orchestra.run.smithi063:> sudo ceph --cluster c2 config set global mon_health_to_clog false 2021-03-24T21:25:03.745 INFO:teuthology.misc:Shutting down mds daemons... 2021-03-24T21:25:03.792 INFO:teuthology.misc:Shutting down osd daemons... 2021-03-24T21:25:03.792 DEBUG:tasks.ceph.c2.osd.0:waiting for process to exit 2021-03-24T21:25:03.793 INFO:teuthology.orchestra.run:waiting for 300 2021-03-24T21:25:03.880 INFO:tasks.ceph.c2.osd.0:Stopped 2021-03-24T21:25:03.881 DEBUG:tasks.ceph.c2.osd.1:waiting for process to exit 2021-03-24T21:25:03.882 INFO:teuthology.orchestra.run:waiting for 300 2021-03-24T21:25:03.945 INFO:tasks.ceph.c2.osd.1:Stopped 2021-03-24T21:25:03.946 DEBUG:tasks.ceph.c2.osd.2:waiting for process to exit 2021-03-24T21:25:03.946 INFO:teuthology.orchestra.run:waiting for 300 2021-03-24T21:25:04.034 INFO:tasks.ceph.c2.osd.2:Stopped 2021-03-24T21:25:04.034 INFO:teuthology.misc:Shutting down mgr daemons... 2021-03-24T21:25:04.035 DEBUG:tasks.ceph.c2.mgr.x:waiting for process to exit 2021-03-24T21:25:04.035 INFO:teuthology.orchestra.run:waiting for 300 2021-03-24T21:25:04.062 INFO:tasks.ceph.c2.mgr.x:Stopped 2021-03-24T21:25:04.063 INFO:teuthology.misc:Shutting down mon daemons... 2021-03-24T21:25:04.063 DEBUG:tasks.ceph.c2.mon.a:waiting for process to exit 2021-03-24T21:25:04.064 INFO:teuthology.orchestra.run:waiting for 300 2021-03-24T21:25:04.145 INFO:tasks.ceph.c2.mon.a:Stopped 2021-03-24T21:25:04.146 INFO:tasks.ceph:Checking cluster log for badness... 2021-03-24T21:25:04.147 DEBUG:teuthology.orchestra.run.smithi063:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2021-03-24T21:25:04.209 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-0 on ubuntu@smithi063.front.sepia.ceph.com 2021-03-24T21:25:04.210 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/c2-0 2021-03-24T21:25:04.685 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-1 on ubuntu@smithi063.front.sepia.ceph.com 2021-03-24T21:25:04.685 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/c2-1 2021-03-24T21:25:05.001 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c2-2 on ubuntu@smithi063.front.sepia.ceph.com 2021-03-24T21:25:05.001 DEBUG:teuthology.orchestra.run.smithi063:> sync && sudo umount -f /var/lib/ceph/osd/c2-2 2021-03-24T21:25:05.157 INFO:tasks.ceph:Archiving mon data... 2021-03-24T21:25:05.159 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/mon/c2-a to /home/teuthworker/archive/sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi/5994655/data/c2.mon.a.tgz 2021-03-24T21:25:05.159 DEBUG:teuthology.orchestra.run.smithi063:> mktemp 2021-03-24T21:25:05.166 INFO:teuthology.orchestra.run.smithi063.stdout:/tmp/tmp.VGmVRkRXwV 2021-03-24T21:25:05.167 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar cz -f - -C /var/lib/ceph/mon/c2-a -- . > /tmp/tmp.VGmVRkRXwV 2021-03-24T21:25:05.625 DEBUG:teuthology.orchestra.run.smithi063:> sudo chmod 0666 /tmp/tmp.VGmVRkRXwV 2021-03-24T21:25:05.689 DEBUG:teuthology.orchestra.remote:smithi063:/tmp/tmp.VGmVRkRXwV is 2MB 2021-03-24T21:25:05.818 DEBUG:teuthology.orchestra.run.smithi063:> rm -fr /tmp/tmp.VGmVRkRXwV 2021-03-24T21:25:05.827 INFO:tasks.ceph:Cleaning ceph cluster... 2021-03-24T21:25:05.827 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2021-03-24T21:25:05.830 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /etc/ceph/c2.conf /etc/ceph/c2.keyring /home/ubuntu/cephtest/c2.data /home/ubuntu/cephtest/c2.monmap /home/ubuntu/cephtest/../*.pid 2021-03-24T21:25:05.880 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2021-03-24T21:25:05.891 INFO:tasks.ceph.ceph_manager.c1:Canceling any pending splits or merges... 2021-03-24T21:25:05.892 DEBUG:teuthology.orchestra.run.smithi052:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster c1 osd dump --format=json 2021-03-24T21:25:06.176 INFO:teuthology.orchestra.run.smithi052.stdout: 2021-03-24T21:25:06.177 INFO:teuthology.orchestra.run.smithi052.stdout:{"epoch":48,"fsid":"91c6d487-aa59-4b0e-bf1a-56b2e36b57d7","created":"2021-03-24T20:44:43.350088+0000","modified":"2021-03-24T21:05:43.716730+0000","last_up_change":"2021-03-24T20:44:48.410807+0000","last_in_change":"2021-03-24T20:44:48.410807+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":15,"max_osd":3,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2021-03-24T20:44:47.385777+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2021-03-24T20:44:52.038659+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"13","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"default.rgw.buckets.data","create_time":"2021-03-24T20:45:32.436742+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"16","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":4,"pool_name":"default.rgw.buckets.index","create_time":"2021-03-24T20:45:33.874030+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":5,"pool_name":".rgw.root","create_time":"2021-03-24T20:45:41.215240+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":6,"pool_name":"default.rgw.log","create_time":"2021-03-24T20:45:42.477457+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":7,"pool_name":"default.rgw.control","create_time":"2021-03-24T20:45:44.450931+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"25","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":8,"pool_name":"default.rgw.meta","create_time":"2021-03-24T20:45:46.497099+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"28","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":9,"pool_name":"test-zone1.rgw.buckets.data","create_time":"2021-03-24T20:46:03.450646+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"31","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":10,"pool_name":"test-zone1.rgw.log","create_time":"2021-03-24T20:46:05.657799+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":11,"pool_name":"test-zone1.rgw.control","create_time":"2021-03-24T20:46:07.470858+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"35","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":12,"pool_name":"test-zone1.rgw.meta","create_time":"2021-03-24T20:46:09.500300+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"38","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":13,"pool_name":"test-zone1.rgw.otp","create_time":"2021-03-24T20:46:38.985959+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}},{"pool":14,"pool_name":"test-zone1.rgw.buckets.index","create_time":"2021-03-24T20:48:57.435573+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":8},"application_metadata":{"rgw":{}}},{"pool":15,"pool_name":"test-zone1.rgw.buckets.non-ec","create_time":"2021-03-24T21:05:41.302955+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"48","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}}}],"osds":[{"osd":0,"uuid":"c13babb7-2f09-4437-9256-5b6291c9e646","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6800","nonce":12674},{"type":"v1","addr":"172.21.15.52:6801","nonce":12674}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6802","nonce":12674},{"type":"v1","addr":"172.21.15.52:6803","nonce":12674}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6806","nonce":12674},{"type":"v1","addr":"172.21.15.52:6807","nonce":12674}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6804","nonce":12674},{"type":"v1","addr":"172.21.15.52:6805","nonce":12674}]},"public_addr":"172.21.15.52:6801/12674","cluster_addr":"172.21.15.52:6803/12674","heartbeat_back_addr":"172.21.15.52:6807/12674","heartbeat_front_addr":"172.21.15.52:6805/12674","state":["exists","up"]},{"osd":1,"uuid":"eafd947e-a358-4639-90ca-82fa477f5c89","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6808","nonce":12675},{"type":"v1","addr":"172.21.15.52:6809","nonce":12675}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6810","nonce":12675},{"type":"v1","addr":"172.21.15.52:6811","nonce":12675}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6814","nonce":12675},{"type":"v1","addr":"172.21.15.52:6815","nonce":12675}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6812","nonce":12675},{"type":"v1","addr":"172.21.15.52:6813","nonce":12675}]},"public_addr":"172.21.15.52:6809/12675","cluster_addr":"172.21.15.52:6811/12675","heartbeat_back_addr":"172.21.15.52:6815/12675","heartbeat_front_addr":"172.21.15.52:6813/12675","state":["exists","up"]},{"osd":2,"uuid":"0ccf2543-f2e4-452e-ae1e-743c78a54921","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":46,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6816","nonce":12698},{"type":"v1","addr":"172.21.15.52:6817","nonce":12698}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6818","nonce":12698},{"type":"v1","addr":"172.21.15.52:6819","nonce":12698}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6822","nonce":12698},{"type":"v1","addr":"172.21.15.52:6823","nonce":12698}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.52:6820","nonce":12698},{"type":"v1","addr":"172.21.15.52:6821","nonce":12698}]},"public_addr":"172.21.15.52:6817/12698","cluster_addr":"172.21.15.52:6819/12698","heartbeat_back_addr":"172.21.15.52:6823/12698","heartbeat_front_addr":"172.21.15.52:6821/12698","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:44:46.830510+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:44:46.766548+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2021-03-24T20:44:46.875880+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2021-03-24T21:25:06.190 DEBUG:teuthology.orchestra.run.smithi052:> sudo ceph --cluster c1 config set global mon_health_to_clog false 2021-03-24T21:25:06.489 INFO:teuthology.misc:Shutting down mds daemons... 2021-03-24T21:25:06.490 INFO:teuthology.misc:Shutting down osd daemons... 2021-03-24T21:25:06.490 DEBUG:tasks.ceph.c1.osd.0:waiting for process to exit 2021-03-24T21:25:06.491 INFO:teuthology.orchestra.run:waiting for 300 2021-03-24T21:25:06.572 INFO:tasks.ceph.c1.osd.0:Stopped 2021-03-24T21:25:06.573 DEBUG:tasks.ceph.c1.osd.1:waiting for process to exit 2021-03-24T21:25:06.573 INFO:teuthology.orchestra.run:waiting for 300 2021-03-24T21:25:06.636 INFO:tasks.ceph.c1.osd.1:Stopped 2021-03-24T21:25:06.637 DEBUG:tasks.ceph.c1.osd.2:waiting for process to exit 2021-03-24T21:25:06.637 INFO:teuthology.orchestra.run:waiting for 300 2021-03-24T21:25:06.710 INFO:tasks.ceph.c1.osd.2:Stopped 2021-03-24T21:25:06.711 INFO:teuthology.misc:Shutting down mgr daemons... 2021-03-24T21:25:06.712 DEBUG:tasks.ceph.c1.mgr.x:waiting for process to exit 2021-03-24T21:25:06.712 INFO:teuthology.orchestra.run:waiting for 300 2021-03-24T21:25:06.742 INFO:tasks.ceph.c1.mgr.x:Stopped 2021-03-24T21:25:06.742 INFO:teuthology.misc:Shutting down mon daemons... 2021-03-24T21:25:06.743 DEBUG:tasks.ceph.c1.mon.a:waiting for process to exit 2021-03-24T21:25:06.743 INFO:teuthology.orchestra.run:waiting for 300 2021-03-24T21:25:06.804 INFO:tasks.ceph.c1.mon.a:Stopped 2021-03-24T21:25:06.805 INFO:tasks.ceph:Checking cluster log for badness... 2021-03-24T21:25:06.806 DEBUG:teuthology.orchestra.run.smithi052:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/c1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(PG_AVAILABILITY\)' | head -n 1 2021-03-24T21:25:06.864 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-0 on ubuntu@smithi052.front.sepia.ceph.com 2021-03-24T21:25:06.865 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/c1-0 2021-03-24T21:25:07.272 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-1 on ubuntu@smithi052.front.sepia.ceph.com 2021-03-24T21:25:07.272 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/c1-1 2021-03-24T21:25:07.453 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/c1-2 on ubuntu@smithi052.front.sepia.ceph.com 2021-03-24T21:25:07.454 DEBUG:teuthology.orchestra.run.smithi052:> sync && sudo umount -f /var/lib/ceph/osd/c1-2 2021-03-24T21:25:07.733 INFO:tasks.ceph:Archiving mon data... 2021-03-24T21:25:07.734 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/mon/c1-a to /home/teuthworker/archive/sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi/5994655/data/c1.mon.a.tgz 2021-03-24T21:25:07.735 DEBUG:teuthology.orchestra.run.smithi052:> mktemp 2021-03-24T21:25:07.741 INFO:teuthology.orchestra.run.smithi052.stdout:/tmp/tmp.4M77974DiM 2021-03-24T21:25:07.742 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar cz -f - -C /var/lib/ceph/mon/c1-a -- . > /tmp/tmp.4M77974DiM 2021-03-24T21:25:08.165 DEBUG:teuthology.orchestra.run.smithi052:> sudo chmod 0666 /tmp/tmp.4M77974DiM 2021-03-24T21:25:08.234 DEBUG:teuthology.orchestra.remote:smithi052:/tmp/tmp.4M77974DiM is 2MB 2021-03-24T21:25:08.314 DEBUG:teuthology.orchestra.run.smithi052:> rm -fr /tmp/tmp.4M77974DiM 2021-03-24T21:25:08.321 INFO:tasks.ceph:Cleaning ceph cluster... 2021-03-24T21:25:08.321 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2021-03-24T21:25:08.364 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -rf -- /etc/ceph/c1.conf /etc/ceph/c1.keyring /home/ubuntu/cephtest/c1.data /home/ubuntu/cephtest/c1.monmap /home/ubuntu/cephtest/../*.pid 2021-03-24T21:25:08.380 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2021-03-24T21:25:08.381 DEBUG:teuthology.orchestra.run.smithi052:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-03-24T21:25:08.420 DEBUG:teuthology.orchestra.run.smithi063:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-03-24T21:25:08.437 INFO:teuthology.orchestra.run.smithi052.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-03-24T21:25:08.444 INFO:teuthology.orchestra.run.smithi063.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-03-24T21:25:08.453 INFO:tasks.ceph:Archiving crash dumps... 2021-03-24T21:25:08.455 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/lib/ceph/crash to /home/teuthworker/archive/sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi/5994655/remote/smithi052/crash 2021-03-24T21:25:08.457 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-03-24T21:25:08.506 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/lib/ceph/crash to /home/teuthworker/archive/sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi/5994655/remote/smithi063/crash 2021-03-24T21:25:08.507 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-03-24T21:25:08.551 INFO:tasks.ceph:Compressing logs... 2021-03-24T21:25:08.552 DEBUG:teuthology.orchestra.run.smithi052:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-03-24T21:25:08.555 DEBUG:teuthology.orchestra.run.smithi063:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-03-24T21:25:53.597 INFO:tasks.ceph:Archiving logs... 2021-03-24T21:25:53.602 DEBUG:teuthology.misc:Transferring archived files from smithi052:/var/log/ceph to /home/teuthworker/archive/sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi/5994655/remote/smithi052/log 2021-03-24T21:25:53.605 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar cz -f - -C /var/log/ceph -- . 2021-03-24T21:25:59.374 DEBUG:teuthology.misc:Transferring archived files from smithi063:/var/log/ceph to /home/teuthworker/archive/sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi/5994655/remote/smithi063/log 2021-03-24T21:25:59.376 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar cz -f - -C /var/log/ceph -- . 2021-03-24T21:26:07.119 DEBUG:teuthology.run_tasks:Unwinding manager install 2021-03-24T21:26:07.138 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2021-03-24T21:26:07.139 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-03-24T21:26:07.142 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-03-24T21:26:07.196 DEBUG:teuthology.run_tasks:Unwinding manager clock 2021-03-24T21:26:07.210 INFO:teuthology.task.clock:Checking final clock skew... 2021-03-24T21:26:07.210 DEBUG:teuthology.orchestra.run.smithi052:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-24T21:26:07.233 INFO:teuthology.orchestra.run.smithi052.stdout: remote refid st t when poll reach delay offset jitter 2021-03-24T21:26:07.234 INFO:teuthology.orchestra.run.smithi052.stdout:============================================================================== 2021-03-24T21:26:07.234 INFO:teuthology.orchestra.run.smithi052.stdout:+hv01.front.sepi 216.240.36.24 3 u 4 64 377 0.075 -1.128 0.200 2021-03-24T21:26:07.234 INFO:teuthology.orchestra.run.smithi052.stdout:*hv02.front.sepi 163.237.218.18 2 u 5 64 377 0.079 -0.938 0.315 2021-03-24T21:26:07.235 INFO:teuthology.orchestra.run.smithi052.stdout:-hv03.front.sepi 192.207.62.39 3 u 8 64 377 0.078 1.980 0.293 2021-03-24T21:26:07.235 INFO:teuthology.orchestra.run.smithi052.stdout:+hv04.front.sepi 96.245.170.99 3 u 62 64 377 0.071 -0.871 0.219 2021-03-24T21:26:07.236 DEBUG:teuthology.orchestra.run.smithi063:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-03-24T21:26:07.259 INFO:teuthology.orchestra.run.smithi063.stdout: remote refid st t when poll reach delay offset jitter 2021-03-24T21:26:07.259 INFO:teuthology.orchestra.run.smithi063.stdout:============================================================================== 2021-03-24T21:26:07.261 INFO:teuthology.orchestra.run.smithi063.stdout:+hv01.front.sepi 216.240.36.24 3 u 56 128 377 0.093 -0.593 0.068 2021-03-24T21:26:07.262 INFO:teuthology.orchestra.run.smithi063.stdout:*hv02.front.sepi 163.237.218.18 2 u 115 128 377 0.071 -0.047 0.029 2021-03-24T21:26:07.262 INFO:teuthology.orchestra.run.smithi063.stdout:-hv03.front.sepi 192.207.62.39 3 u 58 128 377 0.088 2.387 0.037 2021-03-24T21:26:07.262 INFO:teuthology.orchestra.run.smithi063.stdout:+hv04.front.sepi 96.245.170.99 3 u 112 128 377 0.070 0.141 0.158 2021-03-24T21:26:07.263 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-03-24T21:26:07.275 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2021-03-24T21:26:07.276 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2021-03-24T21:26:07.304 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-03-24T21:26:07.338 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2021-03-24T21:26:07.371 INFO:teuthology.task.internal:Duration was 2815.113117 seconds 2021-03-24T21:26:07.373 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-03-24T21:26:07.404 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2021-03-24T21:26:07.405 DEBUG:teuthology.orchestra.run.smithi052:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-03-24T21:26:07.408 DEBUG:teuthology.orchestra.run.smithi063:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-03-24T21:26:07.458 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2021-03-24T21:26:07.459 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi052.front.sepia.ceph.com 2021-03-24T21:26:07.459 DEBUG:teuthology.orchestra.run.smithi052:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-03-24T21:26:07.470 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi063.front.sepia.ceph.com 2021-03-24T21:26:07.470 DEBUG:teuthology.orchestra.run.smithi063:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-03-24T21:26:07.513 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2021-03-24T21:26:07.514 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-03-24T21:26:07.516 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-03-24T21:26:07.583 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2021-03-24T21:26:07.606 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2021-03-24T21:26:07.607 DEBUG:teuthology.orchestra.run.smithi052:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-03-24T21:26:07.624 DEBUG:teuthology.orchestra.run.smithi063:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-03-24T21:26:07.642 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2021-03-24T21:26:07.656 DEBUG:teuthology.orchestra.run.smithi052:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-03-24T21:26:07.668 DEBUG:teuthology.orchestra.run.smithi063:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-03-24T21:26:07.680 INFO:teuthology.orchestra.run.smithi052.stdout:kernel.core_pattern = core 2021-03-24T21:26:07.699 INFO:teuthology.orchestra.run.smithi063.stdout:kernel.core_pattern = core 2021-03-24T21:26:07.729 DEBUG:teuthology.orchestra.run.smithi052:> test -e /home/ubuntu/cephtest/archive/coredump 2021-03-24T21:26:07.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:26:07.756 DEBUG:teuthology.orchestra.run.smithi063:> test -e /home/ubuntu/cephtest/archive/coredump 2021-03-24T21:26:07.774 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-03-24T21:26:07.775 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-03-24T21:26:07.793 INFO:teuthology.task.internal:Transferring archived files... 2021-03-24T21:26:07.797 DEBUG:teuthology.misc:Transferring archived files from smithi052:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi/5994655/remote/smithi052 2021-03-24T21:26:07.798 DEBUG:teuthology.orchestra.run.smithi052:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-03-24T21:26:07.825 DEBUG:teuthology.misc:Transferring archived files from smithi063:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi/5994655/remote/smithi063 2021-03-24T21:26:07.825 DEBUG:teuthology.orchestra.run.smithi063:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-03-24T21:26:07.849 INFO:teuthology.task.internal:Removing archive directory... 2021-03-24T21:26:07.850 DEBUG:teuthology.orchestra.run.smithi052:> rm -rf -- /home/ubuntu/cephtest/archive 2021-03-24T21:26:07.868 DEBUG:teuthology.orchestra.run.smithi063:> rm -rf -- /home/ubuntu/cephtest/archive 2021-03-24T21:26:07.896 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2021-03-24T21:26:07.911 INFO:teuthology.task.internal:Not uploading archives. 2021-03-24T21:26:07.911 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2021-03-24T21:26:07.938 INFO:teuthology.task.internal:Tidying up after the test... 2021-03-24T21:26:07.940 DEBUG:teuthology.orchestra.run.smithi052:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-03-24T21:26:07.943 DEBUG:teuthology.orchestra.run.smithi063:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-03-24T21:26:07.948 INFO:teuthology.orchestra.run.smithi052.stdout: 524339 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 21:26 /home/ubuntu/cephtest 2021-03-24T21:26:07.951 INFO:teuthology.orchestra.run.smithi063.stdout: 524339 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 21:26 /home/ubuntu/cephtest 2021-03-24T21:26:07.953 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2021-03-24T21:26:08.006 INFO:teuthology.nuke:Checking targets against current locks 2021-03-24T21:26:08.075 DEBUG:teuthology.nuke:shortname: smithi052 2021-03-24T21:26:08.075 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-24T21:26:08.077 DEBUG:teuthology.nuke:shortname: smithi063 2021-03-24T21:26:08.077 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-03-24T21:26:08.097 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-24 20:30:17.140919', 'description': '/home/teuthworker/archive/sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi/5994655', 'locked_by': 'scheduled_sage@teuthology', 'up': True, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdmeEuI/ElAp8wa3lVeLfXYahy/pX0c/EdGRCcDqpaRhyuHrjHU0PEe8ckqbf14SelozcIfJk0CStTIKUBckWCPM5LZwtXt7Vh7NQCk6AHxWQq1egfGetVeWbzuPWS+q+doQ+4EyNnnZyWjnIXPtJc4R4tr/mCnKYWRQqax2ypu7oGzF/XCnsQeiRDdY1LWNxobRIpSIEGCJfD4NgJ1MSC4TrgjAw95hrAZgI480tzXReYyKL+WsdNrH/M4fVdlNXXNn5InUbzLyLYKaUfOkbTzr0Jp6SyzBKkBzcCg7kivZwOCv/HNZuHjh3ToUWZValSR3cjBMiSadmLcUBWcQNN', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi052.front.sepia.ceph.com'} 2021-03-24T21:26:08.100 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-03-24 20:30:17.140899', 'description': '/home/teuthworker/archive/sage-2021-03-24_19:49:41-upgrade:octopus-x-wip-sage-testing-2021-03-24-1139-distro-basic-smithi/5994655', 'locked_by': 'scheduled_sage@teuthology', 'up': True, 'ssh_pub_key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJpxVWP+MN/3YlQybedqnQLiHHtDwzkWm82LhqQYZMkhGsE3UDKFv4wmJiTlzH9DcdOW55ds5k0h8KWkxBwtGUf2Rpz3o1MYiLkHk7T/pSG1CC02Iwy7Mt8Je0N3HebLVwXGmBUDu+VIuo8CZNk3dW2KYmWISpqWkywv3/IsA4aMBtRnpYGISapE0Y67MEsM7thcPSAtroKsTa4rZnYbhySBlufPLGKlHn10I53dR5brooEVN4sfHXKATbKSUmG7DxsJOWhvu3zzR+3RSZ/aS8r40WQE5wdKjMt6n6MASnDB2nX/ztpeZK1EebhPzMoK6EhSudrfaq/cOGBB2y7bTL', 'os_version': '18.04', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'ubuntu', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi063.front.sepia.ceph.com'} 2021-03-24T21:26:08.145 INFO:teuthology.orchestra.console:Power off smithi052 2021-03-24T21:26:08.146 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2021-03-24T21:26:08.169 INFO:teuthology.orchestra.console:Power off smithi063 2021-03-24T21:26:08.171 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2021-03-24T21:26:08.184 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi052.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-03-24T21:26:08.199 DEBUG:teuthology.orchestra.console:pexpect command: ipmitool -H smithi063.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2021-03-24T21:26:08.548 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2021-03-24T21:26:08.549 DEBUG:teuthology.task.console_log:Killing console logger for smithi052 2021-03-24T21:26:08.550 DEBUG:teuthology.task.console_log:Killing console logger for smithi063 2021-03-24T21:26:08.550 DEBUG:teuthology.exit:Finished running handlers